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COMPUTER  NETWORK  RESEARCH 

Advanced  Research  Projects  Agency 
Semiannual  Technical  Report 

June  30,  1972 


1.  INTRODUCTION 

This  Semiannual  Technical  Report  covers  the  period  January  1 
through  June  30,  1972.  Our  activities  have  concentrated  on  computer 
systems  studies  and  computer-communication  network  studies.  -If*  Section 
2  below we  describe  the  results  from  modeling,  analysis  and  measurement 
of  various  aspects  of  computer  systems  behavior.  In  Section  3  we  do  like¬ 
wise  for  networks  and  present  some  specific  results  for  the  ARPA  Network. 

2.  COMPUTER  SYSTEMS  STUDIES 

2.1  Time-Shared  Systems 

Our  strong  effort  in  the  area  of  modeling  of  computer  systems  is 
continuing.  In  the  field  of  time -sharing  algorithms,  R.  Muntz  authored 
a  paper  entitled  "Waiting  Time  Distribution  for  Round-Robing  Queueing 
Systems"  [1] ,  and  demonstrated  that  although  round-robin  has  the  desired 
effect  in  yielding  better  service  to  shorter  jobs,  it  also  results  in 
vastly  increased  variance  in  response  time  for  longer  jobs,  as  compared 
with  first-come-first-served  (FCFS) .  This  study  investigated  finite- 
quantum  systems  with  overhead. 

In  previous  reports  we  have  discussed  the  importance  of  networks 
of  queues  as  structures  which  can  more  realistically  model  multiple- 
resource  systems.  R.  Muntz  and  F.  Baskett  (Stanford  University)  are 
currently  preparing  a  report  [2]  on  new  results  in  this  area.  The  start¬ 
ing  point  for  this  work  was  the  work  by  Chandy  on  a  concept  called  local 
balance.  Their  results  include  the  modeling  of  a  network  in  which  there 
are  different  classes  of  customers.  Customers  in  different  classes  may 
have  different  resource  demand  characteristics.  This  is  useful  in  model¬ 
ing  computer  systems  in  that  different  job  mixes  can  be  modeled  and  also 
in  modeling  computer  networks  in  that  routing  of  messages  based  on  source 
and  destination  can  be  modeled.  Their  results  also  permit  various  types 
of  scheduling  disciplines  at  service  centers  in  the  network  (viz.,  FCFS, 

RR  processor  sharing,  LCFS).  Other  generalities  included  in  these  results 
involve  state-dependent  service  rates  and  arrival  rates. 

This  research  on  networks  of  queues  is  an  on-going  effort.  Work 
is  continuing  in  both  extending  the  analytic  results  and  in  validation 
of  the  models.  At  least  one  graduate  student  is  working  on  his  Ph.D. 
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thesis  in  this  area  of  research. 

A  Master's  thesis  by  Walter  Sheets  [3],  supervised  by  L.  Kleinrock, 
has  resulted  in  the  generation  of  a  simulation  language  and  program  for 
evaluating  the  performance  of  and  experimenting  with  a  variety  of  schedul¬ 
ing  algorithms  for  time-shared  systems.  This  "resource"  is  to  become  one 
of  the  standard  programs  available  to  users  over  the  ARPA  Network. 

2.2  Paging  Algorithms 

Reference  [4]  (included  in  this  report  as  Appendix  A)  considers 
various  working  set  replacement  algorithms.  Page  inter-reference  inter¬ 
val  distribution,  average  page  fault  frequency  (the  frequency  of  those 
instances  at  which  an  executing  program  requires  a  page  of  data  or  instruc¬ 
tions  not  in  the  main  memeory),  average  working  set  size  and  inter-page 
fault-time  (time  between  page  faults)  distribution  for  a  simulated  Work¬ 
ing  Set  Replacement  Algorithm  for  three  typical  programs  with  different 
sizes  were  measured  on  the  UCLA  Sigma  Executive  (SEX)  time-sharing  sys¬ 
tem  via  page  reference  strings.  These  measured  results  are  reported  in 
this  paper.  The  average  page  fault  frequency  relationships  between  work¬ 
ing  set  parameters  and  process  scheduling  are  discussed.  These  relation¬ 
ships  are  useful  in  planning  the  working  set  size  and  process  scheduling 
which  optimize  system  efficiency. 

2.3  Buffer  Behavior 

A  paper  entitled  "Buffer  Behavior  from  Mixed  Input  Traffic  and 
Single  Constant  Output  Rate,"  by  Chu  and  Liang  [5]  ,  is  included  as  Appen¬ 
dix  B.  A  queueing  model  with  limited  waiting  room  (buffer),  mixed  input 
traffic  (Poisson  and  compound  Poisson  arrivals),  and  constant  service 
rate  is  studied.  Using  average  burst  length,  traffic  intensity,  and 
input-traffic  mixture  rate  as  parameters,  there  are  obtained  relationships 
among  buffer  size,  overflow  probabilities,  and  expected  message-queueing 
delay  due  to  buffering.  These  relationships  are  portrayed  on  graphs  that 
can  be  used  as  a  guide  in  buffer  design.  Although  this  study  arose  in 
the  design  of  statistical  multiplexors,  the  queueing  model  developed  is 
quite  general  and  may  be  useful  for  other  industrial  applications. 

2.4  Miscellaneous 


Estrin,  Muntz  and  Uzgalis  published  a  paper  "Modeling  and  Measure¬ 
ment  and  Computer  Power"  [6],  which  is  included  as  Appendix  C.  This  paper 
provides  an  informal  definition  of  computer  power  and  three  applications 
of  the  definition  to  issues  which  will  influence  our  ability  to  influ¬ 
ence  computer  systems  in  the  1970's.  For  the  purposes  of  the  paper,  a 
computer  system  is  composed  of:  a  centralized  hardware  configuration;  a 
set  of  terminals  for  entry  and  exit  of  user  programs  and  data;  and  users 
and  user  protocol  for  entry  and  exit.  There  is  no  accepted  measure  for 
global  power  or  performance  of  computer  systems.  There  is  even  no 
accepted  measure  for  computer  cost.  Only  when  a  subsystem  or  subfunction 
is  isolated  does  it  become  possible  to  determine  key  parameters.  However, 
it  is  useful  to  hypothesize  such  measures  and  consider  influences  on  them. 


In  this  way,  the  first  section  provides  a  context  for  the  other  sections 
by  reviewing  parameters  which  make  computing  systems  more  or  less  power¬ 
ful.  The  remaining  three  sections  of  the  paper  are  applications  of  the 
definition.  The  second  section  gives  a  critique  of  the  state  of  model¬ 
ing.  The  third  section  characterizes  measurement  tools.  The  fourth 
secticn  discusses  the  role  of  measurement  at  the  user  interface. 

A  paper  entitled  "Fisheye:  A  Lens-Like  Computer  Display  System," 
by  L.  Kleinrock  and  K.  Stevens,  to  be  published  in  the  Communications  of 
the  ACM  [7] ,  considers  the  potential  of  a  new  computer  display  system. 

This  system  permits  global  vision  as  well  as  local  magnification  simul¬ 
taneously  and  has  shown  to  be  quite  effective  in  scanning  large  data  sets. 

3.  COMPUTER-COMMUNICATION  NETWORKS 

3. 1  Analysis 

A  paper  by  Frank,  Kahn  and  Kleinrock  entitled  "Computer  Communi¬ 

cation  Network  Design  --  Experience  with  Theory  and  Practice"  [6]  is  in¬ 
cluded  as  Appendix  D.  The  design  of  the  ARPA  Computer  Network  brought 
together  many  individuals  with  diverse  backgrounds  and  philosophies.  In 
this  paper,  methods  used  in  the  design  of  the  Network  are  reviewed  from 
the  vantage  of  over  two  years  experience  in  its  development.  The  design 
variables,  system  constraints,  and  performance  criteria  for  the  network 
are  discussed  along  with  an  evaluation  of  the  tools  used  to  design  an 
efficient  and  reliable  system.  The  design  procedures  and  the  conclusions 
reached  about  the  network's  properties  appear  to  be  generally  applicable 
to  message  switched  networks.  Consequently,  the  results  of  this  paper 
should  be  useful  in  the  design  and  study  of  other  store-and- forward  com¬ 
puter  communication  networks. 

Work  by  Cantor  and  Gerla  [9]  has  resulted  in  the  acceptance  of 
their  paper,  "The  Optimal  Routing  of  Messages  in  a  Computer  Network  via 
Mathematical  Programming,"  and  gives  a  computationally  efficient  exact 
algorithm  for  solving  an  important  claco  of  problems.  The  problem  of 
finding  the  optimal  routes  for  messages  in  a  message-switched  computer 
network  can  be,  under  proper  assumptions,  formulated  as  a  nonlinear 
multicommodity  flow  problem.  Many  techniques  that  solve  the  most  gener¬ 
al  cases  can  be  found  in  the  mathematical  programming  literature;  these 
techniques,  however,  prove  to  be  computationally  inefficient  for  the 
design  of  a  computer  network.  For  that  reason,  considerable  effort  has 
been  spent  in  the  past  in  developing  heuristic  techniques.  Quite  satis¬ 
factory  results  have  been  obtained  and  the  computational  efficiency 
has  been  greatly  improved;  however,  all  of  these  techniques  have  various 
limitations.  This  paper  presents  an  exact  method  which,  by  using  decom¬ 
position  techniques  and  taking  advantage  of  the  particular  formulation 
of  the  problem,  is  computationally  competitive  with  heuristic  methods 
and  is  not  affected  by  their  limitations. 

"Some  Recent  Advances  in  Computer  Communications,"  by  Chu  [10], 
is  included  as  Appendix  E.  Recent  advances  in  computer  communications 
are  discussed,  including:  1)  computer  traffic  characteristics  in  the 
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case  of  short  holding  time  representing  the  inquiry-response  systems; 

2}  telephone  channel  error  characteristics  of  high  speed  voice  band  data 
transmission  on  the  switched  telecommunication  network,  and  of  the  low 
speed  channel  at  a  rate  of  300  bits/sec;  3)  optimal  fixed  block  size  for 
communication  systems  using  error  detection  and  retransmission  as  error 
control  (with  random  or  burst  error  channel);  4)  statistical  multiplex¬ 
ing  (Asynchronous  Time  Division  Multiplexing);  5)  loop  systems;  and 
6)  security  in  computer  communications.  New  areas  needing  further  inves¬ 
tigation  are  included. 

G.  Fultz  completed  his  work  on  "Adaptive  Routing  Techniques  for 
Message  Switching  Computer -Communication  Networks"  [11]  under  the  super¬ 
vision  of  L.  Kleinrock.  This  Ph.D.  thesis  will  be  published  as  one  of 
our  Computer  Systems  Modeling  and  Analysis  Group  reports  and  will  enjoy 
wide  distribution.  This  report  considers  adaptive  routing  techniques 
applicable  to  message  switching  c  omputer-communication  networks  such  as 
the  ART'  Network.  The  emphasis  i  on  the  prediction  of  average  message 
delay  and  the  specification,  implementation  and  evaluation  of  various 
classes  of  message  routing  procedures.  The  fundamental  operational  as¬ 
pects  of  a  message  switching  network  model  are  presented.  This  model, 
which  is  based  upon  the  existing  Al  \  Network,  is  utilized  for  the  form¬ 
ulation  of  theoretical  network  pert jrmance  measures  and  is  the  basis  for 
a  computer  simulation  program  written  to  obtain  the  performance  of  spe¬ 
cific  routing  algorithms.  A  methodology  for  the  investigation  of  message 
routing  strategies  applicable  to  message  switching  networks  is  developed 
and  six  key  areas  requiring  study  rre  identified.  Known  routing  tech¬ 
niques  are  classified  into  three  bro.  J  catagories:  1)  deterministic 
routing  techniques,  2)  stochastic  routing  techniques,  and  3)  flow  con¬ 
trol  routing  techniques.  From  this  classification,  one  can  determine 
which  techniques  are  applicable  to  theoretical  studies  and  which  are  can¬ 
didates  for  operating  network  algorithms.  The  remainder  of  the  report 
is  concerned  with  the  following  investigations:  1)  network  performance 
measures  and  models,  2)  specification,  implementation  and  evaluation  of 
deterministic,  stochastic  and  flow  control  routing  algorithms,  and  3) 
the  impact  of  a  network's  size  and  topology  on  message  routing  procedures. 

3.2  Measurement  Activities 


During  the  period  of  March  1  through  June  30,  1972,  steps  were 
taken  to  accelerate  and  broaden  the  Network  Measurement  effort,  mainly 
through  the  effort  of  V.  Cerf.  Two  major  goals  are: 

a)  the  creation  of  a  network  measurement  laboratory 

b)  execution  of  as  wide  a  variety  of  measurement  experiments 
as  possible 

These  goals  are  not  restricted  to  UCLA's  Network  Measurement  Center,  but 
also  include  other  interested  sites  around  the  network.  In  order  to  co¬ 
ordinate  network-wide  measurement  efforts,  a  Network  Measurement  Group 
was  formed  in  mid-March.  As  chairman  of  this  group,  V.  Cerf  has  sought 
to  initiate  cooperative  measurement  experiments  among  the  network  sites. 
Two  early  projects  arising  from  the  formation  of  this  group  include  the 
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4)  NWC  and  NNMP:  These  programs  permit  a  user  to  sequence  through 
the  data  obtained  by  NETSTAT,  accumulating  data  and  printing  out 
selected  statistics  messages  (or  trace  or  snapshot  messages)  ob¬ 
tained  from  IMP'S.  This  facility  is  essentially  a  low  level  fil¬ 
tering  and  formatting  service. 

5)  TMPROC:  This  program  accepts  as  input  files  from  NETSTAT  and 
produces  statistical  analyses  of  traffic  length  distribution, 
delays  and  transmission  rates,  and  link  utilization  observed  dur¬ 
ing  transmissions  between  TINKER  AFB  and  McCLELLAN  AFB.  The  re¬ 
sults  are  published  separately  for  traffic  from  TINKER  to 
McCLELLAN  and  vice-versa. 

6)  SURVEY:  This  program  automatically  polls  the  HOSTS  in  the  net 
and  collects  status  information,  delay  to  perform  ICP  (if  ICP  is 
possible).  The  resulting  data  are  placed  in  a  file  for  further 
processing  (BY  DMCG,  eventually). 

7)  GRAPHIT:  This  program  utilizes  the  IMLAC  Fortran  graphics  package 
and  permits  us  to  present  our  results  graphically  on  the  two  avail¬ 
able  IMLAC  terminals.  Eventually  this  facility  will  permit  inter¬ 
active  monitoring  of  experiments  while  they  are  being  run.  The 
facility  will  be  helpful  at  the  ICCC  in  October. 

The  network  measurement  effort  has  benefited  from  the  strong  support  of 
the  SPADE  software  staff.  In  particular,  the  installation  of  a  time-of- 
day  clock,  a  high  priority  "super  queue,"  and  a  mechanism  for  scheduling 
interrupts  at  pre-determined  times  have  made  it  possible  to  design  a 
HOST  traffic  generator  for  the  SEX  system.  A  programming  philosophy  is 
taking  shape,  partly  as  a  result  of  J.  Postel's  and  M.  Kampe's  programming 
work.  This  philosophy  advocates  the  use  of  'HELP'  routines  embedded  in 
all  user  level  programs  which  will  respond  to  the  bewildered  user's  cry 
for  help  with  increasingly  detailed  instructions  and  explanations.  The 
mechanism  to  provide  this  service  will  be  common  to  all  programs  and  will 
permit  programmers  to  create  and  modify  their  HELP  sections  through  the 
use  of  the  EDIT  program,  without  the  need  to  recompile  or  reassemble  any 
working  program.  The  ease  with  which  such  help  sections  can  be  provided 
will  contribute  to  the  quality  and  quantity  of  these  facilities. 

ONGOING  PROJECTS 


1)  Specification  of  an  artificial  traffic  generator  for  SDC's  DDP- 
516. 

2)  Installation  of  the  SMOG  package  in  the  SEX  system  (under  the 
direction  of  C.  Maxwell). 

3)  Design  of  the  Message  Switching  Protocol  Instrumentation  (with 
D.  Walden  of  BBN) . 

4)  Preparation  of  papers  for  the  following  conferences: 


* 
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a)  COMPCON  72:  "Selected  ARPANET  Measurement  Experiments" 
with  W.  Naylor. 

b)  WESCON  72:  "Selected  ARPA  Network  Measurements" 
with  W.  Naylor. 

These  papers  will  contain  the  results  of  different  experiments 
performed  on  the  net. 

5)  Preparation  of  a  measurement  demonstration  for  ICCC(in  conjunction 
with  R.  Kahn,  BBN) . 

3.3  Software  Development 

This  period  has  been  one  of  a  high  level  of  involvement  in  ARPA 
Network  protocol  development.  J.  Postel  has  been  particularly  active  in 
Telnet  and  Remote  Job  Entry  protocol  issues  and  has  also  participated  in 
File  Transfer  and  Graphics  protocol  development.  He  co-authored  a  paper 
on  "Function  Oriented  Protocols  for  the  ARPA  Computer  Network"  [13]  which 
is  attached  as  Appendix  G.  Furthermore,  the  UCLA-NMC  implementation  of 
the  user-Telnet  program  has  been  augmented  by  adding  facilities  to  send 
and  receive  data  from/to  files.  This  is  a  very  flexible  arrangement  and 
is  further  explained  in  a  section  of  the  SEX  Notebook  (the  system  refer¬ 
ence  manual)  titled  "The  Telnet  Switch." 

4.  CONCLUSIONS 


This  period,  then,  has  seen  some  exciting  advances  in  the  modeling 
of  computer  systems,  particularly  in  the  creation  of  the  general  model  for 
queueing  networks.  The  ARPA  Network  measurement  effort  has  accelerated 
considerably  and  numerous  measurements  have  been  Reported  already;  ongoing 
experiments  continue  at.  present.  The  analytical  progress  in  networks  is 
also  moving  along  rapidly  and  new  results  are  forthcoming.  The  symbiosis 
between  modeling,  analysis  and  measurement  continues  to  be  a  healthy  one, 
and  we  offer  our  progress  as  a  prime  example  of  the  mutual  benefit  one 
achieves  in  such  a  relationship. 
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ABSTRACT 


Page  Inter-reference  Interval  distribution,  average  page  fault 
frequency  (the  frequency  of  those  Instances  at  which  an  executing  program 
requires  a  page  of  data  or  Instructions  not  in  the  main  memory)  average 
working  set  size  and  inter-page  fault-time  (time  between  page  fault) 
distribution  for  a  simulated  Working  Set  Replacement  Algorithm  for  three 
typical  programs  with  different  sizes  were  measured  on  the  UCLA 
Executive  (SEX)  time-sharing  system  via  page  reference  strings.  These 
measured  results  are  reported  In  this  paper.  The  average  page  fault  fre¬ 
quency  relationships  between  working  set  parameters  and  process  scheduling 
are  discussed.  These  relationships  are  useful  In  planning  the  working  set 
size  and  process  scheduling  which  optimize  system  efficiency. 
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Introduction 


Memory  management  becomes  a  severe  problem  In  multiprogramming  and 
virtual  memory  systems.  In  a  multiprogramming  system,  many  programs  are  con- 
currently  executed  by  the  processor.  Thus  the  main  memory  is  shared  by  many 
programs.  Since  the  total  size  of  all  of  the  programs  far  exceeds  the  size 
of  the  main  memory,  in  order  to  keep  infonnation  that  will  be  used  in  the 


near  future  in  the  main  memory,  the  system  constantly  moves  Information 

between  several  levels  of  storage  media. 

In  this  paper,  we  consider  the  case  of  paged  memory  systems:  that 


is.  the  address  spaces  are  partitioned  into  equal  size  blocks  of  contiguous 
addresses.  The  paged  memory  system  has  been  used  by  many  computer  systems. 
However,  the  basic  page  replacement  problem  of  deciding  which  page  should  be 
kept  in  main  memory  and  which  should  be  removed  when  additional  space  is 
needed  is  still  little  understood  and  has  been  of  considerable  interest. 
Obviously,  the  page  removed  should  be  a  page  with  the  least  probability  of 
being  needed  in  the  near  future.  The  difficulty  lies  in  trying  to  determine 
which  page  this  will  be  without  incurring  difficult  implementation  problems 


at  the  same  time. 

Many  replacement  algorithms  have  been  proposed  and  studied  in  the 
past:  such  as  Random.  First-in  First-out,  Stack  Replacement  Algorithms 
(for  example.  Least  Recently  Used  (LRU)),  and  the  Working  Set  Replacement 
Algorithm. C2:1  The  first  three  replacement  algorithms  require  a  fixed  size 
memory  space  for  each  process.  The  Working  Set  Replacement  Algorithm,  how 
ever,  requires  a  variable  size  storage  space  for  each  process  and  the  size 
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varies  with  program  demands.  This  variable  storage  space  provides  an  adaptive 
capability  In  the  replacement  algorithm  which  Is  quite  appealing.  The  work¬ 
ing  set  principle  of  memory  management  states  that  a  program  may  use  a  pro¬ 
cessor  only  If  Its  working  set  (set  of  pages)  Is  in  the  main  memory,  and  no 
working  set  pages  of  an  active  program  may  be  considered  for  removal  from  the 
main  memory.  Properties  of  the  working  set  replacement  algorithm,  the  rela¬ 
tionships  among  page  Inter-reference  Interval,  average  page  fault  frequency 

and  average  working  set  size  for  the  Working  Set  Replacement  Algorithm  are 

T31 

described  In  a  recent  paper  by  Denning  and  Schwartz. L  J 

Because  of  the  complex  nature  of  program  behavior,  analytical  esti¬ 
mation  of  the  above  mentioned  parameters  of  program  behavior  becomes  very 
difficult.  Yet  this  Information  Is  Important  In  the  planning  of  an  efficient 
replacement  algorithm  that  optimize  system  performance.  Therefore  we  employ 
measurement  techniques  for  such  estimations.  We  collect  data  about  the  pat¬ 
tern  of  references  to  all  the  pages  which  comprise  the  executed  program,  and 

measure  these  parameters  experimentally  via  Interpretive  execution.  This 

[4] 

technique  has  been  used  previously  to  measure  dynamic  program  behavior 

[5] 

and  also  to  measure  the  performance  of  Belady's  Optimal  Replacement  Algorithm 
and  LRU  replacement  algorithms'6*^ 

Here  we  report  the  measured  program  behavior  of  the  Working  Set 
Replacement  Algorithm.  We  shall  first  report  measurement  results  such  as 
page  Inter-reference  Interval  distribution,  average  page  fault  frequency, 
average  working  set  size  and  Inter-page-fault- time  distribution.  We  then 
discuss  the  use  of  average  page  fault  frequency  to  determine  the  working  set 
parameter,  and  propose  a  page  fault  scheduling  algorithm  for  process  scheduling 
which  Improves  system  efficiency. 
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II. 


Measurements  and  Results 


The  working  set  W(t,t)  at  a  given  time  t  Is  the  set  of  distinct  pages 
referenced  In  the  time  Interval  ((t-x+1),  t);where  x  Is  called  the  working 
set  parameter.  The  working  set  size  w(t,x)  Is  the  number  of  pages  In  W(t,x). 
The  average  working  set  size  S(x)  defines  as  S(x)  ■  Jj;^  j w(t,x)|  . 
r'or  systems  employing  working  set  replacement  algorithms,  several  parameters 
of  Interest  are:  1)  page  Inter-reference  Interval  distribution  F(x)  which 
describes  the  fraction  of  the  page  Inter-reference  Intervals  less  than  x; 

2)  average  page  fault  frequency  m(x)  which  describes  the  average  number  of 
page  faults  per  page  reference  for  working  set  parameter  x;  3)  average 
working  set  size  S(x);  and  4)  Inter-page- fault- time  (time  between 
page  fault)  distribution  P(t,x)  which  describes  the  fraction  of  the  inter¬ 
page-fault-times  less  than  or  equal  to  t  for  a  given  x. 

F(x)  Is  a  fundamental  distribution;  It  closely  relates  to  the  other 
three  parameters.  When  we  assume  that  the  page  reference  rate  Is  one  page 
per  unit  time,  we  know  that  the  page  references  that  result  In  page  faults 
are  those  references  whose  Inter- reference  Intervals  exceed  x.  Thus,  m(x)  * 

l-F(x).  It  can  be  shown^  that  S(x)  *  £  m(Z).  Thus,  S(x)  Is  closely 

Z-0 

related  to  m(x).  l/m(x)  Is  the  average  running  time  between  page  faults. 

Since  P(t,x)  Is  the  fraction  of  Inter-page- fault-time  less  than  or  equal  to 

t»  l/m(x)  Is  the  time  average  of  the  density  function  P(t+l,x)  -  P(t,x); 

00 

that  Is,  l/m(x)  *  £  t*[P(t+l,x)  -  P(t,x)]. 
t»l 

To  employ  measurement  techniques  for  estimating  these  parameters,  we 
collect  data  about  the  pattern  of  references  to  all  the  pages  which  comprise 
the  executed  program  and  measure  these  parameters  experimentally  via  inter¬ 
pretive  execution.  For  this  purpose  an  Interpreter  for  the  UCLA  Sigma-7 
time- sharing  system  has  been  developed.  This  Interpreter  Is  capable  of 
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executing  S1gma-7  object  programs  by  handling  the  latter  as  data  and  repro¬ 
ducing  a  program's  sequence  of  references.  This  sequence.  In  turn,  can 
then  be  used  as  Input  to  programs  which  simulate  the  Working  Set  Replacement 
Algorl thm. 

Three  different  programs  with  different  sizes  were  Interpretlvely 
executed,  and  their  behavior  was  Investigated  under  the  Working  Set  Replace¬ 
ment  Algorithm.  A  FORTRAN  Compiler  was  chosen  as  the  representative  for  a 
small  program.  WiTA-7  was  chosen  as  the  representative  for  a  large  program. 
It  translates  programs  written  In  META-7  to  the  assembly  language  of  the 
S1gma-7.  A  DCDL  (Digital  Control  Design  Language)  compiler  was  chosen  as  a 
representative  for  a  medium  size  program.  This  compiler  Is  written  In 
META-7.  DCDL  translates  specifications  of  digital  hardware  and  micro¬ 
program  control  sequences  Into  Interpretive  code. 

Table  1  shows  some  characteristic  properties  of  these  programs. 

The  column  'size'  Is  divided  Into  two  parts.  'Static*  refers  to  the  number 
of  pages  necessary  to  store  the  program  as  an  executable  file  on  a  disk 
where  one  page  consists  of  512  32-bit  words.  'Dynamic'  Indicates  the  number 
of  different  pages  actually  referenced  while  processing  the  given  Input 
data.  The  difference  between  the  number  of  pages  In  static  and  dynamic 
results  from  the  fact  that  programs  creat  new  pages  during  execution  for 
working  storage  areas  and  that  not  all  pages  of  programs  are  reference 
during  executing  a  specific  set  of  Input  data. 
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Table  1.  Program  sizes 

of  the 

three  measured  programs 

Size 

Number  of  page  references 

Static 

Dynamic 

FORTRAN 

24 

34 

1,000,000 

DCOL 

44 

58 

1,000,000 

META- 7 

84 

153 

1,000,000 

Figure  1  shows  the  average  page  fault  frequency  iji(t)  for  the  three 
programs.  We  note  that  all  three  programs  exhibit  similar  page  fault  char¬ 
acteristics.  The  average  page  fault  frequency  decreases  rapidly  with  x. 

Large  programs  tend  to  have  a  slower  rate  of  decrease.  The  reason  for  such 
characteristics  Is  mainly  the  locality  of  the  program;  that  Is,  during  any 
Interval  of  execution,  a  program  favors  a  subset  of  its  pages,  and  this  set 
of  favored  pages  changes  its  membership  slowly.  Further,  the  locality  for 
large  programs  Is  usually  larger  than  thtt  of  small  programs.  The  page 
Inter- reference  Interval  distribution  F(x)  »  l-m(x)  can  be  obtained  easily 
from  m(x).  The  average  working  set  sizes  as  a  function  of  x  are  shown  in 
Figure  2.  Measurement  data  support  the  premise  that  average  working  set  size 
Increases  as  program  size  Increases  and  reaches  a  constant  level  as  x  reaches 
a  certain  value.  The  P(t,x)‘s  of  the  three  programs  for  selected  x's  are 
shown  in  Figure  3.  We  note  that  P(t,x)  is  very  sensitive  to  x  and  program 
size.  For  a  given  program,  the  average  inter-page- fault- time  Increases  as  x 
Increases.  This  occurs  because  for  the  small  x  case,  many  of  the  pages 
to  be  referenced  in  the  near  future  are  in  the  secondary  memory; 
thas  the  average  working  set  size  Is  very  small  and  yields  a  high  page  fault 
rate.  For  the  large  x  case,  most  of  the  pages  are  in  the  main  memory  which 
yields  a  large  average  working  set  size  and  a  small  page  fault  rate.  For 
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a  given  x,  large  size  programs  have  a  higher  page  fault  rate  than  that  of 
a  small  size  program.  In  the  next  section  we  shall  discuss  the  applications 
of  these  parameters  to  determine  the  working  set  parameters  and  process 
scheduling  which  improve  system  efficiency. 

III.  Applications  of  Measurement  Data 
A.  Working  Set  Parameter  x 

Working  Set  Parameter  x  is  an  important  parameter  which  affects 
page  fault  rate,  memory  utilization,  and  thus  system  efficienty.  The  measure¬ 
ment  data  support  the  fact  that  x  should  be  chosen  according  to  the  execut¬ 
ing  program  (e.g.,  locality)  and  system  organization  (e.g.,  available  memory 
size  and  the  speed  ratio  between  main  and  secondary  memory).  If  x  is  not 
properly  chosen,  for  example  if  x  is  too  short,  then  pages  are  removed  from 
the  main  memory  while  still  potentially  useful.  This  results  in  high  page 
traffic  between  the  different  levels  of  memory.  If  x  is  too  long,  then 
pages  that  are  not  needed  may  remain  in  the  main  memory,  which  is  an 
inefficient  use  of  memory  space.  Instead  of  choosing  x  arbitrarily,  we 
propose  to  determine  x  from  the  measured  m(x)  and  designate  it  as  x  .  As 
a  result,  x°  is  now  closely  related  to  program  behavior  as  well  as  to  system 

organization. 

The  efficiency  of  a  program  is  defined  as  the  ratio  of  total 
virtual  running  time  to  total  real  running  time  (total  virtual  time  and  total 
page  waiting  time);  that  is. 
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Eff  .  total  virtual  running  time 
total  real  running  time 

0) 

*  1 
T+m(x)R 


where  R  ■  A/T 

A  »  Access  time  of  the  main  memory 
T  *  Access  time  of  the  secondary  memory 

Since  R  Is  fixed  for  a  given  system,  from  (1)  we  know  a  fixed  average  page 
fault  frequency  m(x)  Insures  a  certain  level  of  efficiency. 

Suppose  we  would  like  the  system  to  operate  at  an  average  page 
fault  level  of  about  l£f4  page  faults/ reference;  that  is,  one  page  fault 
In  every  104  page  references.  Then  from  Figure  1,  t°  for  Fortcomp,  DCDL 
and  META-7  are  22,  45,  and  54  m  sec  (1  usee  per  page  reference)  respectively. 

From  Figure  2,  the  corresponding  average  working  set  size  is  15,  36,  and 
39  pages. 

Usually  In  a  multiprogramming  environment  several  types  of 
programs  may  be  concurrently  operated  by  the  operating  system.  The  working 
set  parameter  of  such  a  system  may  either  be  variable  of  fixed.  In  the 
variable  x  case,  the  x°  should  change  from  one  program  to  another;  while  in 
the  f1xed  T  case’  the  T°  rema*ns  f1xed  for  all  types  of  programs.  Because 
0f  th*  s1mpl1city  of  «  fixed  x  scheme.  It  requires  less  overhead  to  Implement 
than  the  variable  x  scheme.  However,  the  efficiency  may  not  be  as  high  as 
that  of  the  variable  x  case. 

One  way  to  determine  the  value  of  a  fixed  x  Is  to  use  the 
weighted  average  working  set  parameters  of  each  program;  that  is, 
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(2) 


where  x^°  *  working  set  parameter  for  the  1th  program  that 
selected  from  Its  m(x) 

ui  «  relative  usage  frequency  of  the  1th  program 
n  ■  total  number  of  distinct  programs  used  In  the  system 

The  decision  as  to  which  scheme  should  be  used  for  a  given  system 
should  be  based  on  program  behavior,  relative  usage  frequency  of  all  the 

llstlnrt  programs  used  by  the  system,  and  the  overhead  In  Implementing  these 
schemas . 


B.  Process  Scheduling 

In  a  multiprogramming  system,  to  increase  system  efficiency  and 
t°  reduce  response  time  for  short  jobs,  the  job  queues  for  CPU  processing 
usually  have  several  priority  levels.  Let  us  consider  a  system  having  two 
levels  of  queues:  Short  Quantum  Queue  (SQQ)  and  Long  Quantum  Queue  (LQQ). 
SQQ  has  a  higher  priority  than  LQQ.  All  jobs  enter  the  SQQ.  Processes  In 
the  SQQ  are  given  one  time  slice  at  a  time.  The  process  Is  put  at  the  back 
of  the  SQQ  after  the  process  either  Incurred  a  page  fault  or  used  up  the 
time  slice;  that  Is,  the  process  Is  serviced  In  a  round-robin  fashion.  A 
process  stays  In  the  SQQ  until  Its  short  quantum  time  runs  out.  It  Is  then 
put  on  the  front  of  the  LQQ.  The  LQQ  will  not  be  serviced  until  the  SQQ  Is 
empty.  A  process  In  the  LQQ  receives  service  until  Its  long  quantum  time 
runs  out.  It  Is  then  put  at  the  end  of  the  LQQ. 

When  a  system  Is  properly  designed,  such  scheduling  algorithms 
yield:  1)  fast  response  time  to  short  jobs,  and  2)  most  of  the  short  jobs 
are  run  In  the  SQQ  and  long  jobs  (  compute-bound  processes  )  will  run  In  the 
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LQQ.  Since  LQQ  provides  more  memory  space  for  each  process  than  that  of 
SQQ,  such  scheduling  yields  less  page  swapping. 

If  the  quantum  time  of  the  SQQ  Is  too  abort,  then  many  of  the 
short  jobs  will  be  In  the  LQQ;  If  the  quantum  time  Is  too  long,  then  many 
computational  jobs  will  be  In  the  SQQ.  The  system  Is  designed  such  that  most 
of  the  short  jobs  finish  their  processing  In  the  SQQ  and  only  the  compute- 
bound  processes  enter  Into  the  LQQ.  The  short  quantum  time  should  be  larger 
than  the  average  real  process  time  of  short  jobs.  However,  the  process  time 
varies  from  one  process  to  another.  In  addition,  the  processing  time  Is 
further  complicated  by  page  faults  occurring  during  Its  execution. 

The  real  processing  time  of  a  process  Is  the  sum  of  the  virtual 
process  time  and  the  total  time  wasted  due  to  page  faults  of  that  process. 

For  example,  two  processes  requiring  the  same  amount  of  virtual  CPU  orocess- 
Ing  time  could  have  very  different  page  fault  frequences,  and  thus  yield  very 
different  real  processing  time.  Therefore  the  real  processing  time  Is 
extremely  difficult  to  estimate. 

We  know  that  page  fault  frequency  has  great  Influence  on  system 
efficiency  and  the  response  time  of  the  short  jobs.  We  propose  to  use  a 
page  fault  as  a  measure  In  process  scheduling;  that  Is,  when  a  process 
exceeds  a  certain  number  of  page  faults  or  exceeds  the  quantum  time  of  the 
SQQ  (whichever  occurs  first),  then  the  process  switches  from  the  SQQ  to  the 
LQQ.  We  shall  call  such  a  scheme  a  page  fault  scheduling  algorithm.  In  a 
multiprogramming  environment,  the  CPU  Idle  times  due  to  page  swapping  between 
main  and  secondary  memories  are  directly  affected  by  the  page  fault  frequency. 

The  paqa  fault  scheduling  algorithm  should  be  effective  In  reducing  CPU  Idle  time 
And  Improve  system  efficiency.  (See  Appendix). 

*For  a  system  operating  In  a  multi  programming  environment,  wc  should  a! so 
Include  the  time  spent  In  waiting  for  the  availability  of  CPU. 
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Processes  with  high  page  fault  rates  occupied  In  the  main  memory 
greatly  reduce  the  efficient  utilization  of  main  memory.  The  page  fault 
scheduling  algorithm  adaptively  allocates  the  low  page  fault  rate  processes 
In  the  main  memory  and  higher  page  fault  rate  processes  In  the  secondary 
memory.  Thus  such  scheduling  Improves  the  utilization  of  main  mw*fy.  As 
a  result,  this  will  Improve  the  average  response  time  of  the  system.  An 
analogy  to  the  above  scheduling  algorithm  Is  the  well  known  "serving  the 
shortest  job  first"  algorithm  In  queueing  theory  that  results  In  Improve¬ 
ments  In  average  waiting  time;  except  In  our  case  we  have  further  Improved 
the  memory  utilization  efficiency. 

The  number  of  page  faults  occurring  during  processing  before 
switching  a  process  from  a  SQQ  to  a  IQQ  depends  on  the  response  time  required 
the  number  of  processes  operating  concurrently,  the  replacement  algorithm 
used,  and  page  fault  frequency  characteristics.  Further  study  In  this  area 
Is  needed. 

In  order  to  reduce  response  time,  the  quantum  time  of  the  SQQ 
and  LQQ  are  further  divided  Into  many  time  slices.  The  optimal  slzi*  of  time 
slices  Is  another  Important  parameter  that  affects  system  efficiency.  The 
time  slice  should  be  selected  such  that  most  of  the  processes  either  page 
fault  or  become  Inactive  before  running  out  of  the  time  slice.  Since  P(t,x) 
describes  the  Inter-page-fault-time  distribution  of  a  process  for  a  given  t, 
the  time  slice  for  the  Quantum  Queues  can  be  determined  from  P( t ,t ) .  For 
example.  If  we  wish  95%  of  the  time  that  the  process  will  page  fault  before 
running  out  of  the  time  slice  —  that  Is,  only  5%  of  the  time  the  process 
will  run  to  the  end  of  the  time  slice  --  then  from  Figure  3  we  know 


the  time  slices  of  the  LQQ  for  t  *  10  m  sec  are:  28  m  sec  for  the  FORTRAN 
Compiler,  13  m  sec  for  DCOL,  and  12  m  sec  for  META-7.  Time  slices  for  t  *  25 
it.  sec  are:  58  m  sec  for  the  FORTRAN  Compiler,  38  m  sec  for  0C0L,  and  35  m  sec 
for  META-7.  Thus,  the  measured  Inter-page-fault-time  distribution  provides 
a  good  way  to  determtiethe  optimal  time  slices  for  the  Quantum  Queues  which 
avoids  excessive  unnatural  Interrupts  that  degrade  response  times. 

The  page  fault  scheduling  algorithm,  as  well  as  the  selection 
of  the  time  slice  form  Inter-page-fault-time  distribution,  a  *e  quite  general 
and  can  be  applied  to  other  types  of  replacement  algorithms. 


V. _ Conclusions 

Page  Inter-reference  Interval  distribution,  average  working  set  size, 
average  page  fault  frequency,and  Inter-page-fault-time  distribution  for  three 
typical  programs  with  working  set  replacement  algorithms  are  measured  and 
reported.  Measurement  results  support  program  locality  and  the  following 
working  set  properties:  the  average  page  fault  frequency  decreases  rapidly  as 
Increases  and  Increases  as  program  size  Increases.  Based  on  these  measured 
data,  wcrking  set  parameter  and  process  scheduling  may  be  selected  from  and 
based  on  the  average  page  fault  frequency.  The  time  slices  for  the  Quantum 
Queues  may  be  determined  from  Inter-page- fault- time  distributions.  A  page 
fault  scheduling  algorithm  Is  proposed  for  process  scheduling  In  a  multl- 
progranlng  environment.  Such  an  algorithm  Is  effective  In  reducing  CPU  Idle 
time  and  Improve  system  efficiency. 


♦The  three  measured  programs  are  not  short  jobs;  they  should  be  run  In  LOO 

stems  p(t-i),s  ~  *•«  w 
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Although  the  Working  Set  Algorithm  provides  an  upper  bound  on  replace¬ 
ment  algorithm  performance,  the  high  cost  of  Implementation  prevents  It  from 
being  widely  used.  Therefore  future  research  should  be  In  developing  low  cost 
hardware  devices  for  economically  Implementing  the  Working  Set  Algorithm  or, 
perhaps  even  more  fruitful.  In  developing  new  replacement  algorithms  that 
have  performance  comparable  to  that  of  the  Working  Set  Algorithm  but  are  much 
easier  to  Implement.  For  example,  we  have  recently  studied  a  Page  Fault 
Frequency  Replacement  Algorithm.  Such  an  algorithm  adjusts  the  LRU  (Least 
Recently  Used  )  stack  according  to  page  fault  frequency.  Preliminary  results 
already  Indicate  It  has  excellent  performance. 
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APPENDIX 

A  Cyclic  Queueing  Model  to  Study  CPU  and  I/O  Operations 


To  Illustrate  the  relationships  among  CPU  Idle  time,  average  page 
fault  frequency  and  swapping  time  (time  to  bring  In  a  new  page  from  the  aux¬ 
iliary  memory)  T,  a  cyclic  queueing  model^  Is  used  to  study  CPU  and  I/O 
operations.  The  system  In  Figure  4  consists  of  two  classes  of  service  facili¬ 
ties.  Service  facility  class  I  represents  a  single  CPU;  Its  service  rate  is 
directly  determined  by  the  average  page  fault  rate*  X.  Service  facility  class 
II  represents  k  parallel  I/O  servers  with  each  having  an  average  service  rate 
y  *  |  .  The  k  parallel  servers  represent,  for  example,  a  paging  drum  with  k 
different  sectors.  Using  such  I/O  facilities,  a  high  degree  of  overlap  of  I/O 
requests  can  be  achieved  In  a  multiprogramming  system  with  relatively  lew  page 
fault  frequency. 

Let  pij  be  the  probability  that  a  job  leaving  server  1  will  proceed 
to  server  j.  Me  assume  that  the  job  leaves  CPU  (server  0)  and  goes  randomly 

to  the  k  I/O  servers  for  service;  thus  poj  a  F  »  for  J  *  1 »  2 . k.  Since 

jobs  which  have  finished  their  I/O  operations  always  return  for  CPU  operations, 
P-IQ  ■  1  for  1  ■  1,  2,  ...,  k;  and  all  the  other  P-jj's  are  equal  to  zero. 

Let  N  be  the  total  number  of  jobs  In  the  system,  and  let  n^  denote 
the  number  of  jobs  In  service  plus  the  number  in  queue  at  the  l*^1  server.  The 

state  of  the  system  can  then  be  determined  by  the  k  +  1  tuple  (nQ,n1 . nk) 

In  which  y  ni  *  N.  The  number  of  distinguishable  states  of  the  system — equal 
to  the  number  of  partitions  of  N  customers  among  k  +  1  servers- -is  (^). 

*for  a  system  using  Working  Set  Replacement  Algorithm  with  parameter  t,  then 
X  ■  m(x). 
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Let  P(nQ,n-| . .  ,nk)  be  the  stationary  probability  that  the  system  is 
in  state  (nQ,n.|,...,nk),  and  let  all  the  service  times  be  assumed  to  be  ex¬ 
ponentially  distributed.  Then  the  steady  state  equations  can  be  written  in 
the  form: 


k 

e(n0)  X  +  £  e(nj)  p 

J-l 


p<n0*nl 


I  • 


k 

*  ^  £(^j)  ^  P0j  p(^o^  *^1 ’*  *  *  **  *  *  *^k^ 

j-l 


k 

^  ^  pio  p^o”^ '^1  **  *  *  *^i"^  *  *  *  * 

1«1 


(Al) 


where  the  indicating  function 


e(nj) 


if  Hj  ■  0 

if  "j  *  0 


accounts  for  the  impossibility  of  any  customer  leaving  the  jth  server  If  that 
server  Is  empty. 


The  left  hand  side  of  (Al)  represents  the  rate  of  transition  out 
of  state  (nQtnjf.tn^) ;  and  the  right  hand  side  is  the  rate  of  transition  Into 
this  state.  Solving  (Al)  by  a  method  of  separation  of  variables^ ,  we  have 

1  k 

p(n0»n-| «... ,nk)  »  -|J 

1-1 


(A2) 


where  a  »  X/p  and  the  normalizing  function  G(N)  Is  determined  from  the  fact 
that  the  sum  of  all  the  P(nQ,n1 .... ,nk)  is  equal  to  1.  Thus 
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G(N) 
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k 

£  n.,*N 
1*0  1 


k 

1 T 
1-1 


SE 


N-nQ+k-1 

k-1 


(A3) 


/N-n0+k-l\ 

where  y  k-1  y Is  the  number  of  distinguishable  partitions  of 

k  I/O  servers. 

The  probability  that  the  CPU  Is  Idle  Is 


N-ng  jobs  among 


P0  *  D  ]  p(0vn^ thgt* • • th^) 
k 

£  n.,*N 
1*1  1 

.  1  /N+k-l\/a\N 

"lW\k-l^irj  (A 

_  L  N 

For  the  case  k  *  1 ,  then  (A4)  reduces  to  PQ  *  ^ - 

For  the  case  N  *  3  and  k  *  6,  the  values  of  Pg's  for  selected  a's 
are  shown  In  Table  II. 
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Table  II 

P0  vs.  ■ 

a 

!o 

0.25 

0.003 

0.50 

0.019 

1.00 

0.091 

1.50 

0.187 

2.00 

0.278 

2.50 

0.362 

3.00 

0.431 

3.50 

0.488 

4.00 

0.537 

4.50 

0.577 

5.00 

0.612 

We  note  that  a  Is  the  ratio  of  average  page  swapping  time  (from 
secondary  memory)  to  average  Inter-page-fault- time.  A  large  a  Implies  large 
page  swapping  time  or  small  Inter-page-fault-tlme  (high  page  fault  frequency), 
or  both.  Thus  the  probability  of  CPU  Idle  time  increases  as  a  Increases. 
Hence,  the  page  fault  scheduling  algorithm  should  be  effective  In  reducing 
CPU  idle  time  and  should  thus  Improve  system  efficiency. 
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WORKING  SET  PARAMETER  T,(msec) 


Figure  1.  Average  page  fault  frequency  m(x)  as  a  function  of 
working  set  parameter  x. 


AVERAGE  WORKING  SET  SIZE  S(tMPaGES) 


WORKING  SET  PARAMETER  r,(msec) 

Figure  2.  Average  working  set  size  S(t)  as  a  function  of 
working  set  parameter  t. 
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INTER -PAGE- FAULT-TIME  t,(msec) 

Figure  3.  Inter-Page-Fault-Time  Distribution 


r  -  5  msec 


INTER  -  PAGE-FAULT-  TIME  t,(msec) 

Figure  3.  Inter-Page-Fault-Time  Distribution 
c.  Meta  7  Compiler 


Figure  4.  A  Cyclic  Queueing  system  for 

modeling  CPU  and  I/O  operations 


r 


39 


APPENDIX  B 

buffer  behavior  for  mixed  input  traffic  and  single 

CONSTANT  OUTPUT  RATE 
by  Wesley  W.  Chu  and  Leo  C.  Liang 
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traffic  intensity,  and  input-traffic  mixture  rata  aa  parameters,  we 
obtain  relation  skips  among  buffer  site,  overflow  probabilities,  and 
erpected  message-queueing  delay  due  to  buffering.  These  relation¬ 
ships  are  portrayed  on  graphs  that  can  bo  uaed  aa  a  guide  in  buffer 
design.  Although  this  study  arose  in  the  design  of  statistical  multi¬ 
plexors,  the  queueing  model  developed  ia  quite  general  and  may  be 
useful  for  other  industrial  applications. 


I  INTRODUCTION 

In  many  engineering  problems  such  as  computer-storage 
allocation,  data  compression,  and  data  communication  [1], 
buffer  design  is  one  of  the  important  considerations.  Birdsall 
et  ai.  [2],  and  later  Dor  [3]  have  analysed  buffer  behavior 
with  Poisson  input  arrivals  and  constant  output  rate.  Chu 
[4]  has  studied  the  buffer  behavior  of  a  similar  model  with 
multiple  synchronous  constant  output  rates.  He  has  further 
studied  buffer  behavior  for  batch  Poisson  arrivals  and  a  single 
constant  output  rate  [5],  In  many  data  communication  sys¬ 
tems,  input  traffic  is  a  mixture  of  bursts  (string  of  characters) 
and  single  characters.  For  example,  in  a  computer  communi¬ 
cation  system,  the  cathode-ray-tube  terminal  outputs  are  in 
bursts  and  the  teletypewriter  outputs  are  >n  characters.  Buffer 
behavior  with  such  mixed  input  traffic  is  studied  in  this  paper 

For  a  given  mixed  input  traffic  and  a  constant  output  rate, 
we  are  interested  in  1)  the  relationship  between  overflow 
probability  (the  average  fraction  of  the  total  number  of  arriv¬ 
ing  characters  rejected  by  the  buffer)  and  buffer  siie  at  various 
traffic  intensities,  and  2)  the  expected  queueing  delay  due  to 
buffering.  These  relationships  are  obtained  by  a  technique 
similar  to  Chu  [5).  The  results  in  this  paper  represent  a 
generalisation  of  his  work. 


II.  Analysis  or  BumH  Behavior 

Let  us  define  the  time  to  transmit  a  character  on  the  multi¬ 
plexed  line  as  a  unit  service  interval.  The  input  traffic  arriving 
at  the  buffer  is  assumed  to  be  a  mixture  of  single-character 
inputs  and  burst  (string  of  characters)  inputs.  The  single- 
character  input  X  is  assumed  to  be  Poisson  distributed,  with 
a  rate  A,  characters  per  unit  service  interval  as  shown  in  (1). 

M*)  -  exp  (-X,),  *  =  0,  1,  2,  •  •  •  .  (1) 

The  characteristic  function  for  /,(*•)  is 


qnr(u)  -  exp  [  —  A,  -f  X,  exp  (iu)].  (2) 

For  burst  input  traffic,  we  assume  the  length  /  of  the  burst  )'  is 
Reprinted  by  permiuiou  from  geometrically  distributed  with  mean  l  -  1/9,  and  the  number  of 

IEEE  TRANSACTIONS  ON  COMMUNICATIONS  bursts  Z  arriving  during  a  unit  service  interval  is  Poisson  distri- 

„  .  Vo1-  COM -20.  No.  2,  April  1972  buted  with  a  rate  X,  bursts  funit  service  interval.  The  distribu- 
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Buffer  Behavior  for  Mixed  Input  Traffic  and  Single 
Constant  Output  Rate 

WESLEY  W.  CHU,  membek,  ieke,  and  LEO  C.  LIANG 

Abstract— A  queueing  model  with  limited  waiting  room  (bailor), 
mixed  inpat  traffic  (Poisson  and  compound  Poisson  arrivals), 
and  constant  service  rate  ia  studied.  Using  tvs  rags  burst  length, 
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MO  -  *0  -  1-1,2,  •••  (3) 

and  Ihe  distribution  of  the  number  of  bursts  arriving  during 
a  unit  service  interval  is 

/*(*»)  -  “7  exp  (-X,),  n  «=  0,  1,  2,  •  •  •  (4) 

The  total  number  of  characters  due  to  burst  inputs  that 
arrive  during  the  time  to  transmit  a  character  on  the  multi¬ 
plexed  line  is  a  random  sum  and  equals 

S  -  £  Y,  (5) 

»-0 

where  Y„  a  random  variable  distribuicd  as  (3),  is  the  num¬ 
ber  of  characters  contained  in  the  ith  arriving  burst  and  Z, 
a  random  variable  distributed  as  (4),  is  the  total  number  of 


1  , 


i  _ 
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bursts  arriving  during  the  unit  service  interval  All  of  these 
random  variables  are  assumed  to  be  statistically  independent 
of  each  other.  It  csn  be  shown  that  4«(u),  the  characteristic 
function  of  5  [5],  is 

#*(u)  -  exp  |-X,  +  X,* 

•  exp  (iu)/[l  -  (1  -  6)  exp  (*u)]| .  (6) 

and  ft(j)  (the  probability  that  exactly  ;  characters  will  arrive 
due  to  burst  arrivals  during  a  unit  service  interval)  has  a 
compound  Poisson  distribution 


I  exp  (— X,),  j  -  0.  (7) 

For  mixed  traffic  of  single-character  inputs  (Poisson)  and 
burst  inputs  (compound  Poisson),  the  probability  that  exactly 
n  characters  arrive  during  a  unit  service  interval  IT.  is  the 
convolution  of  jx(k)  and  /,(j);  that  is, 


Thus,  the  average-character  departure  rate  from  the  buffer 
(carried  load)  0  is  less  than  the  average-character  arrival  rate  at 
the  buffer  (offered  load)  y  =  X,  +  X,/.  The  carried  load  can  he 
computed  from  the  probability  that  the  buffer  is  busy;  that  is, 
0  ■>  1  —  pt,  where  p,  is  the  probability  that  tlie  buffer  is  empty, 
which  can  be  obtained  in  the  exact  manner  as  in  |5). 

The  traffic  intensity  p  measures  the  degree  of  congestion  ami 
indicates  the  impact  of  an  input  traffic  stream  u|ion  the  departure 
stream.  Since  the  offered  load  is  represented  in  a  unit  service 
interval,  p  *■  y  *  X,  ■+■  \,l. 

The  overflow  probability  of  the  buffer  (the  average  fraction  of 
the  total  number  of  arriving  characters  rejected  by  the  buffer)  is 


P. , 


offered  load  —  carried  load 
offered  load 


l  -  fl/y. 


(ID 


Let  a  be  the  input-traffic  mixture  rate  that  describes  the 
percentage  of  the  traffic  contributed  by  compound  Poisson 
arrivals.  Clearly,  1  —  a  is  the  percentage  of  the  traffic  con¬ 
tributed  by  Poisson  arrivals.  Thus, 

a  -  X.I/p 
and 


n.  -  /,(n)  •  /x(n) 

n.  - 1  s^(,r M  t  ft : :) 

-  ey-k  exp(-X.)  +  ^  exp  (_(x>  +  K)]i 

n  -  1,  2,  ••• 


n.  -  exp  [-(X,  +  X.)].  (8) 

The  characteristic  function  for  IT.  is 

*«(«)  -  *s(«)*x(«) 

-  exp  { -(X,  +  X,)  +  X,  exp  (iu) 

+  \.e  exp  (iu)/[l  -  (1  -  9)  exp  (tu)]|.  (9) 

The  time  required  to  compute  IT.  from  (8)  is  dependent 
on  n.  For  large  n  (e.g.,  n  >  1000),  the  computation  time  is 
prohibitive.  Using  the  same  technique  as  [5],  we  compute  n. 
via  the  fast  Fourier  transform  (FFT)  inversion  method  as 
follows: 

1  *f"1 

II,  «  —  £  *»x(r)  exp  (—2wtm/ M), 

M  ,aO 

n  -  0,  1,  2,  •  •  •  ,  M  -  1  (10) 

where 
r  2IIu/M; 

*  (-l)1'*; 

M  total  number  of  points  used  to  represent  4sx(r)  “  total 
number  of  II., 

In  order  to  determine  II,  accurately,  they  are  computed  with 
double  precision  on  the  IBM  360/91  at  the  University  of  Cali¬ 
fornia,  Los  Angeles.  Furthermore,  we  want  to  use  as  many  points 
as  possible  to  represent  4sx(')  I  that  is,  we  want  to  make  M  as 
large  as  possible.  Because  of  the  word-length  limitation  of  the 
computer,  double  precision  provides  15-digit  accuracy.  Therefore, 
when  II,  <  10~",  it  is  set  equal  to  sero.  M  is  selected  such  that 
n,>Jf  <  10"“.  The  M  is  different  for  different  values  of  X„  X., 
and  l. 

Since  the  buffer  has  a  finite  sise  of  N,  an  overflow  will  result 
when  a  character  arrives  at  the  buffer  and  finds  the  buffer  is  full. 


1  -  o  =  X,/p. 

In  the  preceding  analysis,  we  have  treated  each  character 
as  a  unit.  However,  in  computing  the  expected  message  delay 
D  due  to  buffering,  we  should  treat  each  message  as  a  unit. 
The  service  time  is  the  time  required  to  transmit  the  entire 
message.  When  the  buffer  size  N  is  large,  for  a  line  with  a 
constant  transmission  rate,  the  service-time  distribution  is  the 
same  as  the  message-length  distribution  except  scaled  by  a 
constant  transmission  rate  factor.  The  message-length  dis¬ 
tribution  for  the  mixed  input  traffic  of  length  m  is 


/*("») 


where 


K 


x,  +  x. 


8(m)  + 


X, 


X,  +  X, 
m 


6(1  -  er\ 

-  1,  2,  3,  • 


«(m)  - 


|1. 

lo, 


m  =  1 
m  >  1. 


(12) 


When  the  overflow  probability  is  very  small,  a  good  ap¬ 
proximation  for  the  expected  message  delay  can  be  computed 
from  a  queueing  system  with  infinite  waiting  room  [4],  The 
expected  queueing  delay  for  an  M/G/l  (Poisson  arrivals 
/general  service/single  output)  queueing  system  is 


where 


n  _  _Ml£!LL 

2(1  -  p) 


X  =  X,  +  xc 

i?[m*j  =  second  moment  of  /«(m). 
It  can  be  shown  that 


£[m’]  -  ^  [X,  +  xt(2  -  eye7).  (14) 


Substituting  (14)  into  (13),  we  have 


D 


P  ,  a(l  —  1  )p 
2(1  -  p)  r  1  -  p 


character-service-times . 

(15) 
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Fig.  4.  Average  buret  length  I  venue  expected  meeeage  delay  D  lor  traffic  mixture 
rate  a  -  0.5. 

III.  Discussion  of  Results 

The  relationship  of  buffer  length  to  overflow  probability  has 
been  computed  for  selected  traffic  intensities  p,  the  expected 
burst  length  l,  and  input-traffic  mixture  rate  a  as  shown  in 
Figs.  1-3. 

The  overflow  probability  depends  upon  the  N,  p,  l,  and  a.  For 
a  given  buffer  size  N,  the  overflow  probability  increases  as  p, 
l,  and  a  increase.  For  a  given  overflow  probability,  the  required 
buffer  size  increases  as  p,  l,  and  a  increase. 

In  our  analysis,  we  have  assumed  that  the  burst  length  is  geo¬ 
metrically  distributed  and  takes  values  from  one  to  infinity.  In 
practice,  however,  the  maximum  burst  length  is  limited  to  a 
finite  number  of  characters.  Because  of  the  long-tail  effect  of  the 
geometric  distribution,  the  result  obtained  here  will  be  more 
conservative  than  that  of  a  truncated  geometric  distribution. 

When  the  average  burst  length  l  equals  unity  or  when  the 
input-traffic  mixture  rate  a  equals  zero,  the  model  reduces  to  the 
Poisson  arrivals  with  constant  output  rate,  which  has  been 
obtained  by  Birdsall  et  al.  [2],  Dor  [3],  and  Chu  [4],  When  a 
equals  unity,  then  the  model  reduces  to  batch  Poisson  arrivals 
with  a  constant  output  rate,  which  has  been  analyzed  by  Chu  (5J. 
For  given  p  and  l,  the  buffer  size  required  to  achieve  a  desired 
level  of  P„t  is  not  simply  proportional  to  a.  For  a  desired  level 
of  Pot,  Nl  (the  required  buffer  size  for  average  burst  length  l)  is 
much  greater  than  l  •  Nt,  where  Nt  is  the  required  buffer  size  for 
burst  length  of  one  (Poisson  input  arrivals). 

The  expected  message  delay  D  due  to  buffering  (calculated 
from  an  M/G/l  system  with  an  infinite  waiting  room)  depends 
upun  a,  p,  and  l.  For  a  given  p  and  a  given  l  (or  a),  from  (15) 
we  note  that  D  is  linearly  proportional  to  a  (or  t).  This  agrees 
with  our  intuition  that  for  a  given  traffic  intensity,  the  message 
delay  increases  as  the  message  length  increases  and  as  the  amount 
of  buret  input  traffic  increases.  The  relationships  between  l  and 
D  for  a  «  0.5  and  selected  p  are  portrayed  in  Fig.  4. 


IV.  Conclusion 

A  finite  waiting- room  queueing  model  with  mixed  input 
traffic  and  constant  output  rate  has  been  studied.  lor  a 
given  traffic  intensity  and  a  given  input-traffic  mixture  rate, 
buffer  behavior  (in  terms  of  buffer  overflow  probability  and 
average  queueing  delay)  lies  between  that  for  Poisson  input 
arrivals  and  that  for  compound  Poisson  input  arrivals.  When 
the  traffic  mixture  rate  n  approaches  zero,  the  buffer  be¬ 
havior  reduces  to  the  Poisson  input  case.  When  a  equals  one, 
the  buffer  behavior  corresponds  to  the  compound  Poisson 
input  case.  The  numerical  results  for  buffer  behavior  are  por¬ 
trayed  in  graphs  that  are  useful  in  buffer  designs. 
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INTRODUCTION 

Since  the  early  1960s  the  literature* 52  reveals 
increasing  concern  with  effectiveness  of  information 
processing  systems  and  our  ability  to  predict,  influences 
of  system  parameters.  A  recent  survey  paper”  discusses 
methods  of  performance  evaluation  related  to  three 
practical  goals:  selection  of  the  best  among  several 
existing  systems;  design  of  a  not-yet  existing  system; 
and  analysis  of  an  existing  aecessible  system.  The 
classification  of  goals  is  useful,  but  we  can  point  to 
neither  the  models  nor  the  measures  nor  the  measure¬ 
ment  tools  to  allow  reliable  judgments  with  respect  to 
those  three  important  goals  at  this  time. 

We  choose  to  discuss  three  issues  which  do  not  fall 
cleanly  into  Lucas'  categories  hut  which  are  certain 
to  influence  our  ability  to  evaluate  computer  systems 
in  the  1970s.  The  three  issues  are:  effectiveness  of 
models  of  computer  systems;  requirements  to  lx-  met 
by  measurement  experiments;  and  application  of 
modeling  and  measurement  to  the  user  interface  with 
computer  systems. 

The  first  section  provides  a  context  for  the  other 
sections  by  reviewing  parameters  which  make  com¬ 
puting  systems  more  or  less  |>owerful.  The  second 
section  gives  a  critique  of  the  state  of  modeling.  The 
third  section  characterizes  measurement  tools.  The 
fourth  section  discusses  the  role  of  measurement  at 
the  user  interface. 


COMPUTER  POWER 

We  consider  a  computer  system  to  he  composed  of: 
a  centralized  hardware  configuration;  a  set  of  terminals 
for  entry  and  exit  of  user  programs  and  data;  an 
operating  system;  public  programs  and  data  bases; 
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user  programs  and  data;  and  users  and  user  protocol 
for  entry  and  exit. 

There  is  no  accepted  measure  for  global  power  or 
performance  of  computer  systems.  There  is  even  no 
accepted  measure  for  computer  cost.  Only  when  a 
subsystem  or  subfund  ion  is  isolated  does  it  become 
possible  to  determine  key  parameters.  However,  it  is 
useful  to  hypothesize  such  measures  and  consider  in¬ 
fluences  on  them. 

Let  us,  therefore,  define  a  conceptual  measure  which 
we  call  computer  system  power,  l\  as  a  multivariate 
polynomial  function  whose  coefficients  arc  significance 
weights.  We  would,  of  course,  like  to  have  a  set  of 
orthogonal  functions  whose  independent  variables 
correspond  to  measurable  parameters  hut  that,  state 
of  happiness  is  not  apparently  within  reach.  In  an 
attempt  to  exemplify  our  philosophy,  the  authors 
discuss  a  set  of  variables  which  should  influence  P 
keeping  in  mind  that  derivation  of  a  figure  of  merit 
would  require  dividing  P  hy  sonic  measure  of  cost. 

We  intuitively  expect  computer  system  power  to 
increase  if  the: 

•  execution  time  of  any  CPU  instruction  is  decreased 

•  access  time  of  any  memory  subsystem  is  decreased 

•  transfer  rate  to  or  from  any  memory  subsystem  is 
increased 

•  transmission  rate  of  any  buss  structure  is  increased 

•  transfer  rate  to  and  from  any  input  or  output  de¬ 
vice  is  increased 

•  delay  in  resource  availability  is  decreased 

•  error  recovery  time  is  decreased 

•  numlier  of  useful  public  programs  is  increased 

•  performance  of  any  public  program  is  increased 

•  access  time  to  any  public  data  base  is  decreased 

•  arrival,  execution,  and  departure  rates  of  user 
programs  are  increased 

•  execution  time  or  resource  requirement  of  any  user 
program  is  decreased 
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•  numlier  of  effective  users  increases 

•  amount  of  protocol  for  any  user  decreases 

In  a  deeper,  even  more  qualitative  sense,  we  expect 
a  computer  system  to  lx*  more  powerful  if  the  following 
conditions  hold: 

•  system  manager  has  a  model  permitting  adaptation 
to  changing  load 

•  errors  and  system  imbalances  are  reported  to 
maintainors  and  developers 

•  program  documentation  and  measurements  permit 
modification  with  few  side  effects 

•  average  number  of  user  runs  before  correct  execu¬ 
tion  is  decreased 

•  the  quality  of  any  user  program  increases  in  the 
sense  that  there  is  more  effective  use  of  a  source 
language  on  a  given  computer  system. 

Although  the  above  observations  are  useful  in 
stating  expected  events  of  concern  they  ignore  inter¬ 
actions  between  such  events  and  give  no  indication 
of  weighted  importance  of  the  individual  events.  We 
further  characterize  our  systems  by  the  following 
simple  remarks. 

If  the  time  required  for  every  physical  transition  to 
reach  its  new  stable  state  were  halved,  we  would  expect 
throughput  of  the  system  to  double.  If  only  some  of 
the  events  were  reduced  in  transition  time,  we  could 
no  longer  guarantee  that  there  would  be  a  reduction 
in  computation  time  because  the  scheduling  of  events 
is  a  function  of  starting  and  stopping  times  of  con¬ 
current  processes.  Anti-intuitive  anomalies23'3  arc 
disturbing  but  do  not  keep  us  from  conjecturing  that 
they  occur  only  infrequently.  If  we  neglect  anomalies, 
then  we  cannot  expect  change  in  execution  time  of 
any  one  instruction  or  any  one  routine  or  any  one 
compiler  to  produce  a  decimal  order  of  magnitude 
change  in  a  sensibly  weighted  function  of  the  above 
parameters.  Given  reasonable  measurement  tools  and 
design  of  measurement  experiments  we  conjecture 
that  somewhere  between  10  percent  and  50  percent 
improvement  in  performance  can  be  accomplished  for 
most  systems  by  changes  in  assignment  and  sequencing 
of  resources.  Although  these  percentages  do  not  seem 
dramatic  in  their  impact,  the  absolute  number  of 
dollars  or  number  of  computer  hours  which  would 
become  available  is  far  from  negligible. 

In  contrast  with  the  heuristic  probing  and  tuning 
of  a  given  system,  much  greater  impact  is  possible 
at  the  user  interface  with  a  computer  system  and  by 
advances  in  models,  particularly  validated  models  of 
our  computer  systems.  For  example,  we  would  guess 


that  there  arc  more  than  10  attempts  to  run  a  program 
during  its  development  before  it  runs  once  “correctly.” 
For  complex  programs  the  ratio  of  number-of-eorreet- 
runs  to  mimber-of-runs  can  approach  zero.  Hence,  if 
the  user  interface  can  be  altered  so  as  to  increase  the 
probability  of  a  correct  run,  large  benefits  may  result. 

The  effect  of  model  development,  is  a  more  sophis¬ 
ticated  and  qualitative  issue.  It  is  self  evident  that  to 
the  extent  that  we  can  predict  liehavior  of  even  a 
subsystem  through  modeling,  we  can  hope  to  isolate 
important  parameters  and  the  way  they  affect  per¬ 
formance.  In  fact,  only  through  modeling  efforts  can 
we  generalize  experimental  results  at  one  center  to 
apply  to  many  others.  Furthermore,  it  has  been 
recognized  that  simulation  is  the  most  widely  Used 
tool  in  evaluation  of  systems.  If  simulation  depends 
upon  precise  imitation  of  a  computer  system,  its 
development  cost  is  generally  prohibitive  and  it  is 
fraught  will  all  the  unreliability  associated  with  one- 
shot  development.  Effective  simulation  depends  upon 
validated  approximate  models  of  systems  and  of  user 
programs.  Creation  of  such  strong  models  is  the  most 
difficult  of  our  tasks.  However,  the  very  process  of 
validating  or  invalidating  simplifying  assumptions 
used  in  models  can  lead  to  new  algorithms  and  im¬ 
proved  models.  Margolin,  Parmclee  and  Schatzoff3# 
very  competently  demonstrate  this  effect  in  their 
recent  study  of  frec-storagc  management  algorithms. 

In  this  section  wc  have  taken  cognizance  of  the  fact 
that  there  is  no  simple  (or  even  complex)  formula  for 
computer  performance.  The  reader’s  attention  has 
been  focussed  on  the  last  five  in  the  list  of  factors 
affecting  computer  performance  because  they  offer 
so  much  more  return.  The  following  sections  review- 
work  in  analytic  modeling,  measurement,  and  the  user 
interface. 

CRITIQUE  OF  ANALYTIC  MODELING 

Any  system  design,  any  measurement  project  or 
any  resource  allocation  strategy  is  based  on  some  con¬ 
ception  of  the  environment  in  which  it  operates.  That 
conception  is  i  model.  It  is  beneficial  to  have  such 
models  explicitly  stated  so  that  they  can  be  explored, 
tested,  criticized  and  revised.  Even  better,  though  not 
often  achieved  to  the  extent  desired,  is  a  formal  anal¬ 
ysis  of  the  models. 

Models  and  methods  of  analysis  vary  greatly.  Our 
concern  here  is  with  probabalistic  models  of  systems 
and  processes  and  also  with  discrete  graph  models  of 
programs.  The  goals  of  these  analyses  are  both  insight 
and  quantitative  results  to  influence  the  design  of 
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systems,  resource  allocation  strategics  anil  possibly 
the  design  of  languages. 

While  most  will  argue  that  the  goals  of  such  analyses 
arc  inherently  worthwhile  and  must  be  pursued,  there 
is  widespread  dissatisfaction  with  the  current  state  of 
the  field.  Basically,  there  arc  three  major  areas  of  dis¬ 
satisfaction.  First,  the  models  are  generally  over¬ 
simplified  in  order  to  make  them  mathematically 
tractable.  This  obviously  makes  the  results  questionable 
and  brings  us  to  the  sceond  major  failing  which  is  that 
analytic  results  are  often  not  validated  by  measurement 
or  simulation.  Moreover,  in  cases  where  system  evalua¬ 
tion  studies  are  carried  out,  the  existing  models  do  not 
seem  powerful  enough  to  provide  a  uniform  basis  for 
measurements.  The  third  major  criticism  is  that  most 
of  ihe  literature  on  analytic  modeling  is  a  collection 
of  analyses  of  specialized  models.  This  points  up  the 
lack  of  very  general  powerful  results  which  would 
allow  analysis  to  become  an  engineering  tool.  As  it  is 
now,  each  new  situation  almost  always  requires  a 
separate  analysis  by  an  expert. 

While  the  above  are  substantial  criticisms,  this  is 
not  to  say  that  analysis  has  not  had  its  impact.  We 
ean  cite,  for  example,  the  working  set  model  of  program 
behavior,1*  the  work  on  stack  algorithms,41  studies  of 
time-sharing  and  multiprogramming  system  resource 
allocation  and  analyses  of  I/O  scheduling,4*4* 
the  work  on  data  transmission  systems  and  on 
networks"' “•'*  and  the  work  on  graph  models  of 
programs.40'  **,  **•  **•  *4,  *4,  *• l0,  ** 

Promising  areas  of  research. 

Multiple  resource  models 

Mueh  analytic  work  has  dealt  with  single  resource 
models.  The  reason  for  this  is  clearly  that  most  of  the 
analytic  tools  which  arc  available  apply  to  single  re¬ 
source  environments.  The  computer  system  analyst 
is  typically  not  a  mathematician  developing  new  tools 
but  is  generally  engaged  in  applying  existing  tools. 
Nevertheless,  computer  systems  are  multiple  resource 
systems  and  we  must  learn  to  analyze  such  systems. 

Some  recent  studies  of  multiple  resource  models  of 
computer  systems  have  been  made  using  results  by 
Gordon  and  Newell.*1  The  general  model  considered 
by  Gordon  and  Newell  is  one  in  which  customers  (or 
jobs)  require  only  one  resource  at  a  time,  but  move 
from  one  resource  to  another.  An  example  is  illustrated 
in  Figure  1  for  three  resources. 

The  nodes  in  this  figure  represent  resources  and  the 
arcs  represent  possible  transitions  from  one  resource 
to  another.  When  a  customer  has  finished  at  resource  i 


Figure  1 — Example  network  of  queues  iimkicI 


he  moves  to  (requires)  resource  j  next  with  proba¬ 
bility  Pi,.  The  arcs  are  labeled  with  these  probabilities. 
The  service  time  at  each  resource  is  assumed  to  be 
exponentially  distributed.  This  is  a  closed  system 
meaning  that  the  number  of  customers  in  the  system 
remains  fixed.  Gordon  and  Newell  have  found  ex¬ 
pressions  for  the  equilibrium  distribution  of  customers 
in  service  or  queued  at  each  resource.  This  allows  one. 
for  example,  to  calculate  the  utilization  of  the  various 
resources.  > 

Moore4*  and  Buzcn11  have  applied-  this  model  to 
multiprogramming  systems.  Moore  measured  the  MTS 
system  to  obtain  the  transition  probabilities  and  mean 
service  times  of  the  resources  and  then  used  the  model 
to  estimate  system  parameters  such  as  resource  utiliza¬ 
tions.  The  relatively  close  agreement  to  measured 
system  parameters  leads  one  to  believe  that  the  model 
can  be  used  to  predict  the  effect  of  some  changes  in 
system  configuration.  In  using  the  model  in  this  way, 
one  must  be  careful  that  the  proposed  changes  do  not 
significantly  affect  the  basic  behavior  of  the  customers. 
Buzcn  used  the  same  model  to  gain  insight  into  re¬ 
source  allocation  in  multiple  resource  models  of  com¬ 
puter  systems.  His  studies  include  the  investigation  of 
buffering  and  the  effects  of  paging  algorithms.  Both 
Moore  and  Buzcn  have  used  the  model  to  try  to  give 
a  meaningful  formal  definition  to  the  term  “bottle¬ 
neck.”  It  is  of  interest  that  they  arrive  at  different 
definitions  of  a  bottleneck.  The  reader  is  referred  to 
the  references  for  details. 
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While  the  studies  mentioned  above  are  clearly  ad¬ 
vances  in  the  study  of  computer  system  models  there 
arc  numerous  open  questions.  For  example,  the  model 
does  not  allow  the  representation  of  the  simultaneous 
use  of  several  resources  such  as  memory  and  CPU 
Also  there  is  no  means  for  representing  the  synchroni¬ 
zation  of  events  such  as  a  process  doing  buffered  I/O. 
Another  limitation  is  that  the  customers  in  the  system 
are  assumed  to  have  the  same  statistical  Imhiavor,  i.e., 
the  transition  probabilities  and  service  time  distribu¬ 
tion  arc  the  same  for  all  customers. 


Bounds  and  approximations 

Kvory  evaluation  technique  makes  use  of  approxi¬ 
mations.  These  approximations  may  arise,  for  example: 
in  estimating  the  system  parameters,  user  and  program 
behavior;  or  in  simplifying  the  mode!  of  the  system 
itself.  There  is  clearly  a  tradeoff  between  types  of 
approximations.  By  simplifying  a  model  one  might  be 
able  to  handle  more  general  types  of  user  and  program 
liehavior.  Much  of  the  analytic  work  has  been  con¬ 
cerned  with  exact  mathematical  solutions  to  models 
which  are  themselves  gross  approximations. 

An  area  which  is  beginning  to  be  explored  is  that  of 
approximate  solutions  to  more  general  models.  For 
example,  Gaver  has  used  the  diffusion  approximation 
for  the  analysis  of  heavily  loaded  resources  in  queueing 
studies.10  The  basic  technique  is  to  consider  that  the 
work  arrival  process  is  not  the  arrival  of  discrete  cus¬ 
tomers  requiring  service  but  rather  a  work  arrival  flow. 
This  work  arrival  flow  is  a  continuous  process  with  the 
same  mean  and  variance  as  the  original  process.  Another 
example  of  the  use  of  approximations  is  the  work  by 
Kimblcton  and  Moore  on  the  analysis  of  systems  with 
a  limiting  resource.*4 

It  is  clear  that  the  use  of  any  approximation  requires 
validation  of  the  results.  This  may  take  the  form  of 
comparing  results  with  measurements  of  an  actual 
system,  simulation,  or  obtaining  bounds  on  the  error 
in  results.  Bounds  may  also  be  applied  in  a  different 
manner.  Much  has  been  written  on  the  analysis  of 
time-sharing  scheduling  algorithms  and  their  effects 
on  response  times.  Kleinrock,  Muntz  and  Hsu*4  have 
reported  on  results  which  in  effect  demonstrate  the 
I  rounds  on  response  time  characteristics  for  any  CPU 
scheduling  algorithm  which  does  not  make  use  of  a 
priori  knowledge  of  customers  service  times.  The  iin- 
lK»rtaiicc  of  the  bounds  is  that  one  can  sec  the  limits  of 
the  variation  in  res|)onse  characteristics  that  are 
lwssible  by  varying  the  scheduling  algorithm  and  the 
extent  to  which  these  limits  have  been  approached. 


Program  behavior 

A  major  problem  that  must  Iw  dealt  with  in  any 
evaluation  effort  concerned  with  computer  system*  is 
program  behavior.  Fven  when  using  approaches  such 
as  iH'iiehmarking  or  trace-driven  modeling  there  is  the 
problem  of  selection  of  programs  which  are  in  some 
sense  representative  of  the  total  population  of  programs 
that  will  be  run  on  the  system. 

Studies  of  memory  management  in  particular  have 
had  to  explicitly  include  models  of  program  behavior. 
The  early  work  in  this  area1-12  stressed  very  general 
but  powerful  aspects  of  program  behavior  such  as 
“locality”  and  “working  set.”  More  recent  work  deals 
with  more  explicit  models  of  the  generation  of  reference 
strings  which  assume  more  about  program  behavior 
but  correspondingly  allow  for  more  detailed  analysis.13 
It  is  hoped  that  these  models  will  permit  more  detailed 
studies  of  multiprogramming  and  procedure  sharing. 

It  is  interesting  to  not*'  that  the  bulk  of  this  work 
has  I  teen  directed  toward  finding  models  which  can 
represent  tin*  universe  of  possible  programs.  More 
particular.y,  the  goals  of  this  research  have  been  to 
isolate  parameters  characterizing  program  Itchavior  to 
which  memory  management  is  sensitive  and  to  compare 
the  effectiveness  of  various  memory  management 
strategies.  This  approach  is  in  line  with  a  common 
theme  which  runs  through  mast  of  the  work  on  resource 
allocation  strategies  in  computer  systems.  That  is,  we 
see  most  allocation  strategies  attempting  to  work  well 
over  the  total  population  of  programs  possibly  utilizing 
measurements  of  recent  past  history  of  the  process  to 
predict  the  near  future.  Outside  of  work  arising  from 
graph  models  of  parallel  programs'-*11  very  little  has 
been  done  to  utilize  a  priori  information  about  a  process. 
Many  systems  do  make  a  priori  distinctions  between 
batch  and  interactive  processes.  It  seems  reasonable 
though  that  much  more  information  may  In'  available 
which  would  Ire  useful  in  allocating  resources.  For 
example,  it  has  I  men  suggested  that  the  time-slice  and 
paging  algorithm  parameters  Ire  tailored  to  the  pro¬ 
cess.4*  Use  of  a  prion  information  assumes  that  the 
process  is  available  for  analysis  prior  to  execution.  This 
is  a  valid  assumption  for  production  jobs,  system 
processes,  and  to  some  degree  for  all  jobs  at  compile 
time.  Since  tin's**  processes  consume  a  significant  por¬ 
tion  of  th<>  system  resources,  gains  in  efficiency  in 
managing  such  processes  might  result  in  major  gains 
in  total  efficiency.  There  are  many  o])en  problems 
associated  with  this  approach : 

I.  Is  there  a  conflict  with  a  program  design  goal  of 
program  modularity?  How  is  information  alnmt 
separately  compiled  procedures  to  be  combined? 
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2.  Should  processes  l>e  permitted  to  advise  the 
system  as  to  their  resource  needs?  How  does  the 
system  protect  itself  against  false  information? 

If.  How  to  manage  resources  effectively  for  pro¬ 
cesses  which  provide  a  priori  information,  and 
also  for  processes  without  associated  a  priori 
information? 

4.  What  kind  of  a  priori  information  is  actually 
useful  to  management  of  a  system:  how  costly 
is  it  to  obtain  and  utilize  effectively? 

5.  How  predictable  arc  the  resource  requirements 
of  processes? 

While  this  approach  has  received  only  some  slight 
mention  in  the  literature,  it  ap|H«ars  to  be  a  fertile 
area  for  research. 

Graph  models  of  programs  provide  an  abstraction 
of  program  structure  governing  flow  of  control  and 
demand  for  resources.1'10  "  *8  They  |>ermit  a  represen¬ 
tation  fitting  somewhere  between  the  full  detail  of 
actual  programs  and  parametric  or  stochastic  repre¬ 
sentations  of  them.  Most  work  using  graph  models 
has  been  concerned  with  concurrent  processing.  How¬ 
ever,  the  graph  model  analyses  explicitly  reveal  sets  of 
independent  tasks  which  liecome  candidates  for  al¬ 
ternate  sequencing  in  sequential  systems. 

Ideally,  we  search  for  models  of  systems  nnd  program 
l>ehavior  which  provide  principles  guiding  synthesis  of 
configurations  along  with  well  founded  resource  manage¬ 
ment  strategies.  Measurement  must  validate  effective¬ 
ness  of  such  strategies.  The  diversity  of  computations 
further  demands  tlint.  measured  parameters  l>e  pro¬ 
vided  to  operating  systems  and  users  in  order  to  permit 
adaptation  to  dynamic  variations  in  system  behavior 
and  to  unavoidable  anomalies  in  systems  and  languages. 

Studies  during  the  latter  half  of  the  '60s  showed  how 
little  attention  had  lieen  given  to  measurability  in 
the  man-made  universe  of  computer  systems.  The 
next  section  characterizes  some  of  the  problems  in 
measurement. 


MEASUREMENT  OF  INFORMATION 

PROCESSING  SYSTEMS 

Tools  for  measurement  of  computer  systems  must 
satisfy  all  of  the  following  requirements;  detection  of 
prescribed  events,  recording  of  detected  events; 
retrieval  of  accumulated  records;  data  reduction;  and 
display. 

We  comment  on  each  in  turn. 


Detection 

W’e  start  by  rejecting  the  absurdity  of  observing  all 
of  the*  states  of  a  system  under  observation  since  it 
would  imply  detecting  the  state  of  every  input,  even 
memory  element  and  every  output  every  time  there 
was  a  change,  along  with  the  time  at  which  the  change 
occurred.  lienee,  any  set  of  measurement  tools  nm.-t 
include  means  of  selecting  a  subset  of  system  states. 

Hard  wan*  measurement  tools  provide  n  prescribed 
mnnlicr  of  sensing  probes  which  may  be  physically 
placed  on  selected  register  or  buss  (mints  in  a  machine 
under  observation.  Measurement  system  registers 
along  with  programmed  comparator*  and  basic  logical 
operations  permit  further  filtering  by  allowing  detec¬ 
tion  of  a  subset  of  the  events  sensed  by  the  prolies. 
liven  with  such  filtering  the  rate  of  change  of  detected 
states  may  be  excessive.  If  the  n'.‘|K)iise  time  of  hard¬ 
ware  measurement  elements  is  insufficient,  basic 
circuit  changes  would  be  required  to  make  the  measure¬ 
ment  feasible.  If  bandwidth  is  insufficient,  it  is  some¬ 
times  possible  to  introduce  a  sampling  signal  anil 
thereby  further  reduce  the  number  of  detected  events. 
In  the  ubsenee  of  interaction  irilli  software  monitor  pro¬ 
grams,  a  hardware  monitor  is  clearly  limited  in  its  utility. 
To  lie  convinced  of  this,  one  need  only  consider  the 
kind  of  program  status  information  change  which  is 
observable  by  prol>e.s  only  when  it  apjienrs  in  the  form 
of  an  operand  during  the  course  of  computation.  Hard¬ 
ware  detection  can  have  the  virtue  of  introducing  no 
artifact  into  the  measured  system  ami  of  lining  able  to 
detect  events  whose  states  are  not  accessible  to  meas¬ 
urement  programs.  Sampled  detection  may  be  made 
more  effective  by  allowing  interference  with  the  ob¬ 
served  process.  If  a  sampling  signal  enforces  a  proper 
interruption,  observed  data  may  l*c  sequentially  sensed 
by  detection  circuits.  The  recently  reported  "Neurol  ron" 
monitor1  is  the  most  interesting  implemented  hardware 
monitor,  and  its  design  shows  the  foresight  of  enabling 
interaction  with  software  monitor  programs. 

Software  measurement  tools  consist  of  programs 
which  detect  selected  events  by  virtue  of  their  insertion 
at  state-change  (mints  in  the  sequential  computational 
process. 'h47-11  This  detection  process  introduces  arti¬ 
fact  in  execution  time,  in  space  requir'  d  for  measure¬ 
ment  program  storage,  and  sometimes  (c.g.,  synchroni¬ 
zation  with  asynchronous  cyclic  processes)  in  qualita¬ 
tive  side  effects  on  existing  computational  processes. 
In  a  sampling  mode,  measurement  programs  can  have 
their  in-line  artifact  reduced  by  disturbing  the  flow 
of  computation  only  at  a  sampling  time.  At  a  sampling 
time,  measurement  programs  may  Ik-  brought  in  to 
check  as  large  a  set  of  memory  states  as  is  needed  and 
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then  control  is  returned  to  the  oljserved  system.  In 
the  absence  of  hardware  Mip|tort,  a  softwnre  monitor 
is  iiimtcMl  to  oliscrvation  of  those  system  states  which 
have  affected  memory  contents  In  one  case,  careful 
analysis  of  measurement  of  PL  I  functions  of  nil  IBM 
360  91‘*  revenled  anomalies  in  recorded  system  stati-s 
which  can  best  be  characterised  as  nrtifnet  introduced 
by  OS  360  when  it  inserts  code  associated  with  I/O 
interrupts  into  the  code  living  measured. 

It  has  lieeome  clear  that  we  are  not  faced  with  mu¬ 
tually  exclusive  alternative's  <f  hnrdwarc  detection 
tools  or  software  deti'ction  tools.  Hnther  bow  much  of 
each;  how  they  are  integrated;  nnd  bow  they  nre  made* 
available  to  ex|ierimenters.  A  pn|ier  by  Nemeth  ami 
Kovner"  presents  a  pleasing  cxnmplc  of  tlu1  |Kiwer  of 
combined  hardware  and  soft  ware  in  the  hands  of  a 
user.  They  point  out  that  facility's  introduced  for  hard¬ 
ware  debugging  are  often  the  kind  useful  in  Inter  program 
measurements. 


Data  Reduction 

The  amount  and  kind  of  data  reduction  is  determined 
by  the  goal  of  the  measurement  ex|M-riment  and  limi'a- 
tions  of  measurement  tool  capabilities  in  detection, 
n •cording  nnd  preparation  for  retrrcvnl  For  example, 
assume  t lint  we  want  to  obtain  n  history  of  utilization 
of  routines  in  order  to  decide  which  should  Is*  kept  in 
primary  storage  ni.d  which  should  lie  kept  in  backup 
storage  Assume,  further,  that  every  time  a  routine  is 
called:  the  name  of  the  called  routine,  the  time  of  day. 
and  the  nnine  of  the  user  is  recorded.  It  would  not  In- 
very  meaningful  to  generate  only  a  history  showing  the 
times  at  which  each  lontinc  in  the  system  was  used  by 
each  user.  Data  reduction  would  be  required  to  deter¬ 
mine.  for  example,  the  total  iiuiiiIht  of  such  uses,  mi 
ordering  of  routines  |>y  mnnlier  of  uses,  n  determina¬ 
tion  of  the  iiiiiiiIht  of  routines  involved  in,  sny  ,V) 
(icrceiit,  of  the  uses  and  their  names,  etc. 


Recording  ■ 

If  an  event  of  interest  has  liven  detected,  its  oc¬ 
currence  must  affect  memory  contents.  Such  net  ion 
may  lie  as  simple  as  incrementing  a  counter  or  as  com¬ 
plex  as  storing  a  lot  of  state  information  for  Inter 
nnalysis.  In  .he  case  ol  nondisturbing  hardware  mea¬ 
surements,  ‘  Xtemal  storage  must  lie  provided  and  (he 
transfer  rate  must  lie  able  to  keep  up  with  the  rate  of 
change-of-state  information  observed  by  a  set  of  prolies 
and  associated  circuits.  In  the  case  of  software  mcasurt*- 
ments,  sufficient  memoiy  space  must  either  lie  pro¬ 
vided  to  record  all  relevant  state  information,  or  dse 
preprocessing  reduction  programs  must  lie  called  in 
to  reduce  storage  requirements. 


Display 

The  goal  of  the  data  reduction  process  is  defined  by 
the  specifud  form  of  display  or  fi-edliaek  to  the  experi¬ 
menter.  If  measurement  is  living  made  for  feedback 
to  nn  operating  system  for  use  in  resonre  location, 
parameter  values  must  lie  delivered  to  memory  cells  to 
lie  accessed  by  the  o|ierating  system.  If  measurement 
is  made  for  accounting  purposes  or.  more  generally,  to 
provide  the  user  with  feedback  aliout  his  quality  of 
use  and  system  rcs|»onse,  results  should  In-  merged  into 
user  nnd  system  mnnager  files.  If  measurement  i-  made 
for  operator  control  nnd  management,  simple  alpha¬ 
numeric  displays  an*  common.  For  e.x|H<riinental  annl- 
ysis  of  system  liehavior,  CRT  displays,  graphs  nnd 
computer  printout  are  generally  required 


Retrieval 


Measurement  Methodology 


In  the  construction  of  any  large  system,  Imth  a  datn 
gathering  and  retrieval  system  must  lie  incorjxiratcd 
into  the  basic  design.  Failure  to  do  so  will  limit  the 
amount  of  instrumentation  avndnble  later  when  ef¬ 
ficiency  questions  arise  For  example,  in  a  large  pro¬ 
gramming  system  which  has  traasient  program  seg¬ 
ments,  data  gathering  is  easily  inserted  into  any 
program  segment;  however,  unless  a  standard  data 
storing  program  is  available,  the  datn  gatheml  cannot 
lie  easily  retrieved.  The  IBM  PL/I  F-lcvel  compiler 
is  an  example  of  a  programming  system  broken  into 
transient  program  segments.  It  fails  to  have  a  data 
storing  program  with  adequate  bandwidth  to  support 
meaningful  measurement  activity. 


The  complexity  of  computer  systems  dictates  s|>ecial 
care  in  the  planning  of  measurement  ex|ierinicnts.  If 
the  results  of  exiierimcnts  an*  not  reproducible,  tliev 
an-  of  little  value.  If  any  assumptions  Hindi  are  not 
recorded  nnd  vnlidatcd,  the  results  cannot  Ih»  general¬ 
ized  nnd  applied  at  another  time  or  plan  A  large  ImhIv 
of  statistical  tlnsiry  is  avnilable  providing  methods  for 
abstracting  pro|icrtics  out  of  individual  data  points, 
but  npplienbility  must  be  cnrefully  checked.  We  have 
little  ho|ie  of  ndhering  to  principles  if  we  do  not  have  a 
measurement  language  to  prescrilie  measurement  ex- 
IH'riini  .its  ns  sequences  of  commented  operations  w  Inch 
are  appropriately  integrated  with  oliscrved  data.  The 
latter  step  needs  wait  upon  creative  development  of 
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measurement  tools  and  their  test  in  meaningful  ex¬ 
periments.  Measurement  capability  must  be  explicitly 
included  during  periods  of  system  design  and  must  bo 
available  for  inclusion  in  user  program  design.  Digital 
computer  systems  and  programs  are  properly  charac¬ 
terized  as  complexes  of  very  elementary  functions. 
Full  systems  or  programs,  therefore,  generally  require 
partition  in  order  to  manage  the  synthesis  process. 
Each  partition  introduces  possible  measures  of  validity 
of  output,  of  performance  and  of  cost.  Means  for  meas¬ 
urement  should  be  checked  at  that  point  in  design  and 
a  value  judgment  made  if  excessive  artifact  would  be 
introduced  by  the  measurement  process. 

If  a  system  contains  the  structure  and  primitive 
operations  satisfying  the  five  requirements  discussed 
in  this  section,  it  carries  the  tools  for  adaptability.  We 
conjecture  that  much  more  than  the  10  percent  to  50 
percent  improvement  alluded  to  in  the  Introduction 
becomes  attainable — particularly  when  measurement 
tools  can  influence  user  behavior. 

COMPUTER  POWER  AND  USER  INTERFACE 

In  the  seventies  some  stronger  effort  must  be  directed 
toward  increasing  computer  power  by  a  reduction  of 
the  complexity  of  the  user  interface. 

Operating  systems  and  Higher  Level  Languages  are 
tools  designed  to  give  the  user  control  of  the  portion  of 
a  computer  system  he  needs.  With  the  exception  of 
work  done  at  SDC,*1'2*  little  reported  effort  has  been 
devoted  to  the  human  engineering  aspects  of  these 
tools.  During  the  last  decade,  while  hardware  made  a 
dramatic  increase  in  power,  the  management  tasks 
required  of  the  operating  system  increased  from  trivial 
to  highly  complex.  At  the  same  time,  the  users  were 
required  to  supply  (in  unnatural  form  like  JCL)  more 
of  the  parameters  which  would  allow  effective  manage¬ 
ment  decisions  to  be  made  by  the  operating  system. 
These  user-supplied  parameters  have  increased  the 
burden  of  complexity  at  the  user  interface — and  re¬ 
duced  the  amount  of  useful  work  a  user  can  accomplish 
in  a  given  period  of  time. 

For  example,  much  of  the  attraction  of  APL  360 
is  its  simplification  of  the  operating  system  interface 
along  with  the  addition  of  immediate  execution  of  its 
concise  powerful  primitives.  A  batch  oriented 
FORTRAN  user  perceives  this  ns  a  tremendous  in¬ 
crease  in  his  computer  power  A  more  sophisticated 
user  might  see  APL  as  a  powerful  desk  calculator  which 
provides  immediate  access  to  functions  similar  to 
functions  he  already  commands,  less  accessibly,  in 
other  languages. 


Another  user  interface  problem  exists  at  the  level 
of  higher  level  languages.  As  more  advanced  hardware 
becomes  available  to  the  user,  lie  seeks  to  solve  more 
complex  problems.  When  a  problem  grow-'  beyond  a 
manageable  point,  the  user  segments  the  problem  into 
pieces  plus  associated  linkages.  In  doing  so,  however, 
he  introduces  a  new  set  of  communication  problems; 
a  change  in  one  program  which  affects  an  interface  can 
now  wreak  havoc  in  another  “completed''  portion  of  the 
problem  solution.  Higher  level  languages  have  been  lax 
in  the  types  of  program  interconnections  (and  inter¬ 
actions)  allowed. 

An  example  of  the  problems  of  creating  a  large  pro¬ 
gramming  system  are  reported  by  Melody  and  Lehman 
using  data  from  the  development  of  OS  360. «  While 
this  study  concerned  programs  written  in  assembly 
language  for  the  IBM  360,  the  properties  which  produce 
the  error  rates  and  modification  ratios  reported  in 
their  paper  are  characteristics  of  all  large  programming 
systems  today. 

Several  techniques  for  improving  the  probabilities 
that  a  program  can  lie  made  error  free  are  available  in 
the  literature.  One  of  the  earliest  is  Dijkstra's  “Notes 
on  Structured  Programming,”"1  am  also  “THE  Pro¬ 
gramming  System."15  His  system  breaks  the  problem 
into  shells  of  “pearls”  or  "primitive"  operations.  Each 
shell  is  built  using  the  “primitives"  of  the  next  lower 
level.  This  system  attempts  to  minimize  interactions, 
forces  the  programmer  to  produce  generalized  func¬ 
tions  which  can  be  tested,  ami  allows  easy  instrumenta¬ 
tion  liecause  of  the  segregation  of  functions. 

Some  disadvantages  of  such  a  hierarchical  scheme 
make  its  practical  application  difficult.  Snell  a  scheme 
increases  initial  development  time  because  it  forces 
developers  to  completely  understand  the  structure  of 
the  system  being  built  and  to  predefine  the  proper 
function  for  each  hierarchy.  Transitions  between  levels 
may  lie  costly.  Functions  at  the  lowest  level  are  the 
most  general  and,  therefore,  the  most  frequently  used. 
Small  inefficiencies  in  these  functions,  or  in  the  method 
of  traversing  levels  of  the  structural  hierarchy,  magnify 
costs  dramatically  and  force  the  user  away  from  cen¬ 
tralized  functions.  This  defeats  the  original  purpose  of 
the  organization. 

Another  disadvantage  of  a  hierarchical  scheme  is 
that  while  instrumentation  of  the  system  is  easy,  in¬ 
terpretation  of  the  measurements  is  generally  not 
Measurement  results  could  change  drastically  if  the 
organization  of  the  program  were  modified.  Therefore, 
it  is  hard  to  tell  how  much  of  what  goes  on  is  due*  to 
the  structural  hierarchy  and  how  much  is  due  to  the 
intrinsic  properties  of  the  program  Such  knowledge 
points  a  way  toward  improvement 
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87 
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A3 
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27 
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25 
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IEM0673 
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12 

IHE320 

8 
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ERROR  DESCRIPTION 


NO  FILE/STRING  SPECIFIED.  SYSIN/SYSPUINT  1IAS 

BEEN  ASSUMED.  _ ftllriVfl 

TEXT  BEGINNING  vyyy  SKIPPED  IN  OU  FOLLOWING 

STMT  NUMBE”  _ 

STATEMENT  Nl  MBI'R  xxxx  HAS  BEEN  DELETED  DUE 
TO  A  SEVERE  ^..KOR  NOTED  ELSEWHERE. 

TEXT  BEGINNING  yyyy  IN  STATEMENT  NUMBER  xxxx 
HAS  BEEN  DELETED. 

DATA  CONVERSION  WILL  BE  DONE  BY  SUBROUTINE 
CALLS. 

OPTION  IN  GET/PUT  IS  INVALID  AND  HAS  BEEN 

DELETED.  ..... 

ILLEGAL  PARENTHESIZED  LIST  IN  STATEMENT  NIM- 
BER  xxxx  FOLLOWS  AN  IDENTIFIER  WHICH  IS  NOT  A 
FUNCTION  OR  ARRAY. 

TEXT  BEGINNING  yyyy  IN  OR  FOLLOWING  STATEMENT 
NUMBER  xxxx  HAS  BEEN  DELETED. 

SEMICOLON  NOT  FOUND  WHEN  EXPECTED  IN  STATE¬ 
MENT  xxxx.  ONE  HAS  BEEN  INSERTED. 

INVALID  USE  OF  FUNCTION  NAME  ON  LEFT  HAND 
SIDE  OF  EQUAL  SYMBOL  OR  IN  REPLY,  KEYTO  OR 
STRING  OPTION. 

ADDRESSING  INTERRUPT. 

FIXED  OVERFLOW . 

FILE  name — END  OF  FILE  ENCOUNTERED. 

ERROR  IN  CONVERSION  FROM  CHARACTER  STRING  TO 

iDITUMPTir 


*  IBM,  PL/I(F)  Programmer *’  Guide  (Appendix  K),  GC28-6594,  January  1971. 


Figure  2a— Moet  frequent  PL/I  errors 
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Figure  2b — Average  persistence  sorted  by  average  persistence 
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NUMBER  OF  ERROR  TYPE* 

ERROR  DESCRIPTION 

ERROR 

OCCURRENCES 

COMPILE-TIME 

143 

SY16 

IMPROPER  ELENIENT(S) 

131 

SYE5 

ILLEGAL  USE  OF  COLUMN  1  ON  CARD 

89 

CGOC 

NO  FILE  SPECIFIED.  SYSIN/SYSPRINT  ASSUMED 

83 

SYOB 

MISSING  SEMICOLON 

76 

SM4E 

ident  HAS  TOO  MANY  SUBSCRIPTS.  SUBSCRIPT  LIST 
DELETED 

55 

SY3A 

IMPROPER  LABEL 

53 

SY04 

MISSING  ) 

48 

SY06 

MISSING  COMMA 

46 

SY09 

MISSING  : 

40 

SM50 

name  NEVER  DECLARED,  OR  AMBIGUOUSLY  QUALIFIED 
(EXPRESSION  REPLACED  OR  CALL  DELETED) 

EXECUTION-TIME 

827 

F.X78 

SUBSCRIPT  number  OF  ident  IS  OUT  OF  BOUNDS 

239 

EX83 

FIXED  POiNT  OVERFLOW 

211 

EXBB 

DELETED  STATEMENT  ENCOUNTERED 

189 

EX7D 

LENGTH  OF  SUBSTRING  LENGTH  OF  STRING 

180 

EX98 

I N COM PATA B LE  OPTIONS  ON  OPEN 

169 

EX7B 

INDEX  OF  SUBSTRING  LENGTH  OF  STRING 

141 

EXB8 

ARRAY  ELEMENT  HAS  NOT  BEEN  INITILIZED.  IT  I< 
SET  TO  0. 

90 

EX9F 

IMPLIED  CONVERSION  NOT  IMPLEMENTED 

46 

EX89 

PROGRAM  IS  STOPPED.  NO  FINISH  ON-UNIT  AFTER 
ERROR 

45 

EXB7 

ident  HAS  NOT  BEEN  INITIALIZED. 

•  Conway,  R.  W.  ct  at,.  User's  Guide  to  PL/C, 
University,  Ithaca,  August  I,  1971. 

The  Cornell  Compiler  for  PL/I,  Release  6,  Department  of  Computer  Science,  Cornell 

Figure  3a  -Most  frequent  PL/C  errors 


Present  studies  of  forced  program  structure  and 
program  proofs  of  correctness  may  begin  to  provide 
models  on  which  HLL  designers  may  base  their  pro¬ 
posals.  However,  any  major  changes  should  be  de¬ 
signed  to  improve  the  user’s  system  so  that  eaeh  pro¬ 
gram  submittal  can  be  a  learning  experience.  In  this 
way  a  programming  system  can  be  called  upon  to  point 
out  unusual  events;  draw  the  programmer's  attention 
toward  possible  errors;  and  yet,  not  produce  volumes 
of  output  which  would  be  costly  to  print  and  which  a 
programmer  would  refuse  to  read. 

Work  at  Cornell  toward  producing  compilers  which 
correct  human  failings  rather  than  punish  them  has 
culminated  in  a  highly  functional,  rapid  compiling, 
and  very  permissive  PL/I  student-oriented  compiler 
called  PL/C.44  This  compiler  does  spelling  correction  of 
keywords,  automatic  insertion  of  missing  punctuation, 
etc.  In  addition  automatic  collection  of  some  statistics 
is  done  at  execution  time.  For  example,  each  label 
causes  a  count  to  be  maintained  of  the  number  of  times 
execution  passed  through  that  labeled  statement. 
Compilers  such  as  these  increase  computer  power  by 
reducing  the  complexity  of  the  user  interface. 

Implementations  of  HLLs  could  further  help  a 


programmer  by  giving  an  optional  cross  reference 
listing  showing  locations  where  a  variable  is  changed, 
could  be  changed,  or  just  referenced.  Items  could  be 
flagged  if  they  were  never  referenced  or  set ;  only  refer¬ 
enced;  or  only  set.  In  the  first  two  cases  spelling  cor¬ 
rection  might  be  applicable.  Statements  which  use 
expensive  library  subroutines  or  other  costly  language 
features  could  be  flagged.  Measurement  nodes  could 
be  easy  to  insert  and  operate.  These  should,  in  turn, 
produce  meaningful  data  which  relate  directly  to 
questions  the  programmer  wanted  to  ask. 

But  such  discussions  have  only  beat  around  the  bush 
itself.  The  real  problem,  the  bush,  is  the  higher  level 
language.  The  real  questions  are:  What  features  are 
error  prone?  What  features  of  the  language  allow  auto¬ 
matic  validity  checking  of  what  is  written?  How  can 
these  properties  be  identified  and  measured?  How  can 
the  knowledge  of  these  things  be  used  to  reduce  com¬ 
plexity  of  the  user  interface  so  that  the  user  perceives 
an  increase  in  his  computer  power?  Which  language 
constructs  arc  seldom  used,  adding  unnecessary  com¬ 
plexity  and  unreliability? 

Efforts  to  measure  the  human  engineering  aspects 
of  computer  language  use37  and  to  provide  feedback 


734  Spring  Joint  Computer  Conference,  1972 


GRAPE  OF  VARlABLt  1C  t AC H  SPACE  »  1C0.00  UNITS. 
SAMPUS  CF  LESS  THAN  5  HAVE  BEEN  DELETED. 


CGOC 

£(76 

EXS9 

SMSO 

exbv 

EX7D 
EX  78 
CC1F 
SV27 
S*<51 

Exfs 

SV  1  6 
SYOA 
S“A7 

£  xda 

EXPB 

SY2F 

SY3e 

SV  1 E 

SY  ID 

SYLb 

SYcb 

SYOE 

SM5E 

SY3A 

SV3A 

SV  1  7 

SVC9 

EX9F 

EX9E 

EXd7 

SY18 

SYIO 

$MAE 

SMA8 

SY1  3 

SY05 

SVE  5 

EX  83 

SYf  2 

SYE  6 

SNAA 

SY3C 

SY3C 

SMA1 

$V;0 


* 

* 


-I . . I . . 1 . . I" 

0.10  G.20  J.  3C  O.AO 


» 


-I _ • - 1 - 1 - 1 - -•  - - 1 - 1 - 1 - ' - I" 

O.SO  0.60  0.70  0.  BO  0.90  1.00 


AVERAGE  PERSISTENCE 

SOATtO  BY  AVERAGE  PERSISTENCE 


Figure  3b 


into  the  design  stages  of  higher  level  languages  and 
the  control  and  command  languages  of  the  operating 
system  may  provide  major  increases  in  computer 
power  by: 

•  increasing  the  number  of  users  who  can  bring 
problems  to  the  machine 

•  decreasing  the  number  of  problem  submissions 
necessary  to  bring  a  job  to  completion 

Work  is  in  progress  at  SDC,M  W  “  moving  toward  a 
man-machine  symbiosis.  These  little  publicized  ap¬ 
proaches  to  measurement  of  human  problem  solving 
and  computer  languages  have  just  begun  to  scratch 


the  surface  of  this  very  important  area.  Work  at  UCLA 
has  attempted  to  identify  properties  of  PL/I  which 
are  prone  to  human  error.  As  a  first  approximation, 
error  rates  and  persistence  curves  of  various  errors 
identified  in  students'  use  of  the  IBM  PL  I  F-lo\ el 
compiler30  is  presented  in  Figure  2.  Corresponding 
results  for  errors  found  by  Cornell  s  PL/C  compiler 
are  presented  in  Figure  3.  Figure  2a  shows  a  table  or 
the  number  of  occurrences  of  the  most  frequent  PL  I 
error  types  recorded  during  compilation  and  during 
execution  times.  Figure  2b  displays  the  persistence  of 
errors  by  PL/I  type  during  the  student  runs.  The 
vertical  coordinate  is  the  error  type  ordered  by  the 
magnitude  of  PERSISTENCL  RATIO.  The  hori- 
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zontal  coordinate  is  the  PERSISTENCE  RATIO  and 
was  calculated  as  an  average  of  (number  of  sequential 
trials  during  which  the  particular  error  persisted) 
divided  by  the  total  number  of  trials.  If  an  error  type 
did  not  occur  in  at  least  5  problem  assignments  it  was 
arbitrarily  deleted  to  keep  the  displayed  range  of 
values  reasonable.  Figures  3a  and  3b  display  the  same 
properties  for  assignments  using  PL/C.  A  total  of  128 
problem  assignments  completed  by  28  students  are 
included  in  the  statistics.  Follow-up  work  is  intended 
to  lead  more  deeply  into  language  design  and  hopefully 
into  new  techniques  for  automatically  localizing  errors 
in  a  program. 

The  basic  technique  for  doing  this  is  to  allow  the 
programmer  to  specify  more  information  than  the  HLL 
processor  needs  to  compile  the  program.  An  example 
would  be  identifiers  of  the  form  “CONSTANT’’  in  a 
PL/I  data  attribute  syntax.  CONSTANTS  as  opposed 
to  variables,  would  only  be  set  by  an  initial  attribute 
and  would  be  illegal  as  a  left-hand  side  of  an  assign¬ 
ment  or  as  an  argument  of  a  pseudo-variable.  In  ad¬ 
dition,  the  program  could  be  considered  as  having  this 
attribute.  At  several  points  in  a  program  (e  g.,  block 
exit  time)  these  constants  would  be  checked  to  see  if 
their  value  had  changed.  If  any  had,  a  warning  would 
be  printed;  the  correct  value  restored;  and  the  program 
would  continue.  Such  a  new  PL/I  data  type  allows 
automatic  checking  for  consistency  to  localize  errors 
and  yet  is  almost  painless  for  a  programmer  to  use. 
When  a  program  is  debugged,  it  is  easy  to  turn  off  this 
kind  of  checking  for  the  sake  of  more  efficient  per¬ 
formance.  In  a  hardware  environment  like  the 
MULTICS  GE  645,  these  errors  can  be  detected  dy¬ 
namically  when  illegal  accesses  occur. 

Debugging  tools  should  be  designed  into  the  lan¬ 
guage  and  taught  as  part  of  the  language,  because  the 
majority  of  the  time  a  programmer  deals  with  a  lan- 


Figure  4b — Modified  information  llow  to  augment  feedback 
in  it  HLL  job 


guage,  he  is  also  dealing  with  an  incorrect  program. 
Subscript  checking,  trace  information,  validity  checking 
at  periodic  intervals,  time  and  eount  information, 
formatted  displays  of  all  program  information  and 
selective  store  and  fetch  recording  are  the  kinds  of 
things  which  should  be  available  to  the  HLL  pro¬ 
grammer. 

In  addition,  measurement  tools  should  be  im¬ 
mediately  accessible  to  any  user  without  burdening 
others,  so  that  if  questions  of  efficiency  arc  raised  they 
can  be  answered  simply  and  quickly.  Some  of  the 
measurement  tools  which  seem  important  are:  (1)  flow 
charts  or  tables  as  optional  output  which  would  stress 
intermodule  dependencies;  (2)  time  and  count  control 
statements  which  could  be  output,  reset  and  inactivated 
under  program  control,  and  would  create  output  auto¬ 
matically  if  the  program  terminated  abnormally  or 
without  expressly  outputting  the  data  gathered;  (3) 
program  size  should  be  easily  accessible  by  the  program 
dynamically  and  summaries  of  available  space  should 
be  available  at  program  termination. 

In  order  to  increase  the  complexity  of  the  pro¬ 
gramming  problems  which  users  can  handle,  languages 
must  be  allowed  to  accommodate  personal  ways  of 
expressing  concepts.  To  do  this,  at  the  very  minimum, 
new  data  types  should  be  available  for  the  programmer 
to  define  as  well  as  operators  which  use  these  data 
types.  This  begins  to  syntactically  approach  the 
Dijkstra  concepts  and  to  allow  easier  application  of 
hierarchically  structured  programs.  Hopefully  these 
approaches  will  increase  the  user’s  computer  power  by 
making  the  development  of  his  programs  easier. 

The  programming  system  itself  should  be  restruc¬ 
tured  so  that  more  information  is  available  to  a  HLL 
processor.  Figure  4a  shows  the  diagram  of  information 
flow  in  a  usual  batch  oriented  system.  The  source  code 
is  compiled;  the  resulting  object  code  is  passed  to  the 
accretion  step  where  library  or  previously  compiled 
programs  are  added  to  it;  and  the  resulting  load  module 
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is  passed  to  the  execution  phase;  finally,  output  from 
execution  is  passed  back  to  the  user.  To  provide  more 
automated  feedback,  Figure  4b  shows  information 
flow  in  a  system  where  statistical  summaries  of  one 
execution  are  available  to  the  next  compilation.  In 
addition,  the  accretion  step  spends  much  more  of  its 
time  checking  linkage  conventions  and  the  validity  of 
argument-parameter  choices.  This  programming,  sys¬ 
tem  has  an  edit/compile  time  library  which  is  designed 
to  help  make  changes  easy.  For  example,  it  keeps 
“COMMON”  declarations  uniform  (read  EX¬ 
TERNAL  if  yon  are  a  PL/I  programmer)  and  it  also 
uses  information  from  the  compiler  to  point  the  user  at 
syntax  errors  and  information  from  the  execution  phase 
to  point  the  user  at  semantic  errors. 

Such  modifications  can  reduce  errors  and  speed  the 
development  of  programs  by  improving  communica¬ 
tion  between  what  are  now  considered  separate  pro¬ 
gram  steps.  However,  the  most  important  changes, 
across  all  the  proposed  modifications,  arc  those  changes 
which  will  allow  the  programmer  to  receive  only  those 
pieces  of  information  relevant  to  the  level  at  which  he 
is  programming  (i.e.,  making  changes).  This  would 
provide  dynamic  help;  help  where  the  programming 
language  acts  as  an  extension  of  the  users’  mind  to 
assist  in  problem  solving  and  optimization. 

It  is  important  to  view  these  changes  which  move 
toward  dynamic  assistance  in  terms  of  costs.  Each 
change  must  cost  something  in  execution  time  over¬ 
head.  Some  of  the  more  powerful  features  like  selective 
fetch  and  store  monitoring  must  be  expensive.  How¬ 
ever,  if  these  features  were  found  valuable,  then  modi¬ 
fication  to  hardware  might  diminish  costs  dramatically. 
Integrating  these  techniques  into  HLLs  must  be 
inherently  costly  because  implementation  and  testing 
of  human  interaction  with  these  diagnostic  features 
are  difficult  to  execute  in  any  controlled  way— much 
work  must  rest  on  subjective  evaluation  of  users’ 
behavior.  Integration  of  aids  into  HLL  translators 
must  be  initially  done  without  those  very  aids  which 
are  deemed  necessary  to  help  programmers  modify 
programs.  Therefore,  any  change  is  fraught  with  risks 
caused  by  lack  of  checks  in  current  systems.  Obviously 
bootstrapping  is  called  for  and  we  can  expect  many 
passes  before  achieving  effective  tools. 

The  development  of  richer  higher  level  languages 
on  the  one  hand,  and  the  development  of  debugging 
services  and  error  correcting  compilers  on  the  other, 
exert  forces  in  the  direction  of  increasing  performance 
at  the  user  interface.  With  appropriate  use  of  models 
and  measurement  much  more  improvement  may  be 
obtained. 


SUMMARY 

Computer  systems  P-re  different  from  other  systems  by 
virtue  of  the  dynamic  fashion  in  which  our  compre¬ 
hension  of  their  behavior  may  be  built  into  their  opera¬ 
tion.  If  validated  models  are  developed,  they  may  then 
be  built  into  the  system  itself  to  serve  adaptive  re¬ 
source  allocation  algorithms.  If  measurement  tools  arc 
effectively  integrated,  they  may  be  made  available  to 
the  user  to  improve  the  quality  of  his  use  of  program¬ 
ming  languages.  If  the  user  is,  in  fact,  a  team  develop¬ 
ing  programming  systems,  the  modeling  and  measure¬ 
ment  facilities  may  serve  to  make  much  more  complex 
programs  possible  because  a  model  of  programs  being 
built  is,  itself,  generally  too  complex  for  a  group  of 
unaided  humans  to  manage  in  an  error  free  way.  In 
the  above  paper  wc  have  sought  to  open  up  these 
questions. 

Wc  have  not  had  much  experience  with  effective 
modeling  and  measurement.  There  is  an  immense 
amount  of  data  to  be  observed  in  a  computer  system 
Cost-effectiveness  of  performance  measurement  must 
be  considered.  As  one  of  our  reviewers  put  it,  “Tins 
reviewer  has  seen  some  measurement  studies  lead  to 
system  improvements  which  will  pay  off  sometime  in 
2018.”  Hopefully  the  1970s  will  see  more  effective 
modeling  and  measurement  introduced  into  the  design 
process  and  selectively  carried  into  developed  systems 
to  he  p  both  interna'  process  management  and  the 
enrichment  of  external  use  through  the  user  interface. 
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INTRODUCTION 

Tho  ARPA  Network  (ARPANET)  project  brought 
together  many  individuals  with  diverse  backgrounds, 
philosophies,  and  technical  approaches  from  the  fields 
of  computer  science,  communication  theory,  operations 
research  and  others.  The  project  was  aimed  at  providing 
an  efficient  and  reliable  computer  communications 
system  (using  message  switching  techniques)  in  which 
computer  resources  such  as  programs,  data,  storage, 
special  purpose  hardware  etc.,  could  be  shared  among 
computers  and  among  many  users.38  The  variety  of 
design  methods,  ranging  from  theoretical  modeling  to 
hardware  development,  were  primarily  employed 
independently,  although  cooperative  efforts  among 
designers  occurred  on  occasion.  As  of  November,  1971, 
the  network  has  been  an  operational  facility  for  many 
months,  with  about  20  participating  sites,  a  network 
information  center  accessible  via  the  net,  and  well  over 
a  hundred  researchers,  system  programmers,  computer 
center  directors  and  other  technical  and  administrative 
personnel  involved  in  its  operation. 

In  this  paper,  we  review  and  evaluate  the  methods 
used  in  the  ARPANET  design  from  the  vantage  of 
over  two  years’  experience  in  th('  development  of  the 
network.  In  writing  this  paper,  the  authors  have  each 
made  equal  contributions  during  a  series  of  intensive 
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discussions  and  debates.  Rather  than  present  merely  a 
summary  of  the  procedures  that  were  use'll  in  the 
network  design,  we  have  attempted  to  evaluate  each 
other’s  methods  to  determine  their  advantages  and 
drawbacks.  Our  approaches  and  philosophies  have  often 
differed  radically  and,  as  a  result,  this  has  not  been  an 
easy  or  undisturbing  process.  On  the  other  hand,  wo 
have  found  our  collaboration  to  be  extremely  rewarding 
and,  notably,  we  have  arrived  at  many  similar  con¬ 
clusions  about  the  network’s  behavior  that  seem  to  be 
generally  applicable  to  message  switched  networks. 

The  essence  of  a  network  is  its  design  philosophy,  its 
performance'  characteristics,  and  its  cost  of  implementa¬ 
tion  and  operation.  Unfortunately,  there  is  no  generally 
accepted  definition  of  an  “optimal”  network  or  even  of 
a  “good”  network.  For  example,  a  network  designed  to 
transmit  large  amounts  of  data  only  during  late  evening 
hours  might  call  for  structural  and  performance  char¬ 
acteristics  far  different  from  one  servicing  large  numbers 
of  users  who  are  rapidly  exchanging  short  messages 
during  business  hours.  We  expect  this  topic,  and  others 
such  as  the.  merits  of  message  switching  vs.  circuit 
switching  or  distributed  vs.  centralized  control  to  be  a 
subject  of  discussion  for  many  years.,>,4  M  M  M'37 

A  cost  analysis  performed  in  1967-196S  for  the*  ARl’A 
Network  indicated  that  the  use  of  message  switching 
would  lead  to  more  economical  communications  and 
better  overall  availability  and  utilization  of  resources 
than  other  methods.38'38  In  addition  to  its  impact  on 
the  availability  of  computer  resources,  this  decision  has 
generated  widespread  interest  in  store-and-forward 
communications.  In  many  instances,  the  use  of  store- 
and-forward  communication  techniques  can  result  in 
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greater  flexibility,  higher  reliability,  significant  tech¬ 
nical  advantage,  and  substantial  economic  savings  over 
the  use  of  conventional  common  carrier  offerings.  An 
obvious  trend  toward  increased  computer  and  com¬ 
munication  interaction  has  begun.  In  addition  to  the 
ARPANET,  research  in  several  laboratories  is  under 
way,  small  experimental  networks  arc  being  built,  and 
a  few  examples  of  other  government  and  commercial 
networks  are  already  apparent.'7  51  •*MI  47  48  •** 

In  the  ARPANET,  each  time-sharing  or  batch 
processing  computer,  called  a  Host,  is  connected  to  a 
small  computer  called  an  Interface  Message  Processor 
(IMP).  The  IMPs,  which  arc  interconnected  by  leased 
50  kilobit/second  circuits, '  handle  all  network  com¬ 
munication  for  their  Hosts.  To  send  a  message  to 
another  Host,  a  Host  precedes  the  text  of  its  message 
with  an  address  and  simply  delivers  it  to  its  IMP.  The 
IMPs  then  determine  the  route,  provide  error  control, 
and  notify  the  sender  of  its  receipt.  The  collection  of 
Hosts,  IMPs,  and  circuits  forms  the  message  switched 
resource  sharing  network.  A  good  description  of  the 
ARPANET,  and  some  early  results  on  protocol  develop¬ 
ment  and  modeling  are  given  in  References  3,  12,  15, 
23  and  38.  Some  experimental  utilization  of  the 
ARPANET  is  described  in  Reference  42.  A  more  recent 
evaluation  of  such  networks  and  a  forward  look  is 
given  in  References  35  and  39. 

The  development  of  the  Network  involved  four 
principal  activities: 

(1)  The  design  of  the  IMPs  to  act  as  nodal  storc- 
and-forward  switches, 

(2)  The  topological  design  to  specify  the  capacity 
and  location  of  each  communication  circuit 
within  the  network, 

(3)  The  design  of  higher  level  protocols  for  the  use 
of  the  network  by  time-sharing,  batch  pro¬ 
cessing  and  other  data  processing  systems,  and 

(4)  System  modeling  and  measurement  of  network 
performance. 

Each  of  the  first  three  activities  were  essentially  per¬ 
formed  independently  of  each  uther,  nl.^rean  the 
modeling  effort  partly  affected  the  IMP  design  effort, 
«A  llisJj!  wlUi  L he  logical 

project. 

The  IMPs  were  designed  by  Bolt  Beranek  and 
Newman  Inc.  (BBN)  and  were  built  to  operate  in¬ 
dependent  of  the  exact  network  connectivity;  the 
topological  structure  was  specified  by  Network  Analysis 
Corporation  (NAC)  using  models  of  network  per¬ 
formance  developed  by  NAC  and  by  the  University  of 
California  at  Los  Angeles  (UCLA).  The  major  efforts 
in  the  area  of  system  modeling  were  performed  at 


UCLA  using  theoretical  and  simulation  techniques. 
Network  performance  measurements  have  been  con¬ 
ducted  during  the  development  of  the  network  by 
BBN  and  by  the  Network  Measurement  Center  at 
UCLA.  To  facilitate  effective  use  of  the  net,  higher 
level  (user)  protocols  are  under  development  by  a 
group  of  representatives  of  universities  and  research 
centers.  This  group,  known  as  the  Network  Working 
Croup,  has  already  specified  a  Host  to  Host  protocol 
and  a  Telnet  protocol,  and  is  in  the  process  of  com¬ 
pleting  other  function  oriented  protocols.47*  We  make 
no  atte  mpt  to  elaborate  on  the  Host  to  Host  protocol 
design  problems  in  this  paper. 

THE  NETWORK  DESIGN  PROBLEM 

A  variety  of  performance  requirements  and  system 
constraints  were  considered  in  the  design  of  the  net. 
Unfortunately,  many  of  the  key  design  objectives  had 
to  be  specified  long  before  the  actual  user  requirements 
could  be  known.  Once  the  decision  to  employ  message 
sw'tching  was  made,  and  fifty  kilobit/second  circuits 
were  chosen,  the  critical  design  variables  were  the 
network  operating  procedure  end  the  network  topology; 
the  desired  values  of  throughput,  delay,  reliability  and 
cost  were  system  performance  and  constraint  variables. 
Other  constraints  affected  the  structure  of  the  network, 
but  not  its  overall  properties,  such  as  those  arising  from 
decisions  about  the  length  of  time  a  message  could 
remain  within  the  network,  the  location  of  IMPs 
relative  to  location  of  Hosts,  and  the  number  of  Hosts  to 
be  handled  by  a  single  IMP. 

In  this  section,  we  identify  the  central  issues  related 
to  IMP  design,  topological  design,  and  network 
modeling.  In  the  remainder  of  the  paper,  we  describe 
the  ma  or  design  techniques  which  have  evolved. 

IMP  properties 

The  key  issue  in  the  design  of  the  IMPs  was  the 
definition  of  a  relationship  between  the  IMP  subnet 
and  thv  Hoals  tu  part  it  mn  responsibilities  so  that 
reliable  and  efficient  operation  would  be  achieved.  The 
k  &  mode  lb  build  1U  UliwUm  nubluff, 
independent  (as  much  as  possible)  of  the  operation  of 
any  Host.  The  subnet  was  designed  to  function  as  a 
“communications  system”;  issues  concerning  the  use  of 
the  subnet  by  the  Hosts  (such  as  protocol  development ) 
were  initially  left  to  the  Hosts.  For  reliability,  the 
IMPs  were  designed  to  be  robust  against  all  line  failures 
and  the  vast  majority  of  IMP  and  Host  failures.  This 
decision  required  routing  strategies  that  dynamically 
adapt  to  changes  in  the  states  of  IMPs  and  circuits, 
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and  an  elaborate  flow  control  strategy  to  protect  the 
subnet  against  Host  malfunction  and  congestion  due  to 
IMP  buffer  limitations.  In  addition,  a  statistics  and 
status  reporting  mechanism  was  needed  to  monitor  the 
behavior  of  the  network. 

The  number  of  circuits  that  an  IMP  must  handle  is  a 
design  constraint-  directly  affecting  both  the  structure 
of  the  IMP  and  the  topological  design.  The  speed  of  tho 
IMP  and  the  required  storage  for  program  and  buffers 
depend  directly  upon  the  total  required  processing 
capacity,  which  must  be  high  enough  to  switch  traffic 
from  one  line  to  another  when  all  are  fully  occupied.  Of 
great  importance  is  the  property  that  all  IMPs  operate 
with  identical  programs.  This  technique  greatly 
simplifies  the  problem  of  network  planning  and  main¬ 
tenance  and  makes  network  modifications  easy  to 
perform. 

The  detailed  physical  structure  of  the  IMP  is  not 
discussed  in  this  paper.4'8  However,  the  operating 
procedure,  which  guides  packets  through  the  net,  is 
very  much  of  interest  here.  The  flow  control,  routing, 
and  error  control  techniques  are  integral  parts  of  the 
operating  procedure  and  can  be  studied  apart  from  the 
hardware  by  which  they  are  implemented.  Most 
hardware  modifications  require  changes  to  many 
IMPs  already  installed  in  the  field,  while  a  change  in 
the  operating  procedure  can  often  be  made  more 
conveniently  by  a  change  to  the  single  operating 
program  common  to  an  IMPs,  which  can  then  be 
propagated  from  a  single  location  via  the  net. 

Topological  properties 

The  topological  design  resulted  in  the  specification  of 
the  location  and  capacity  of  all  circuits  in  the  network. 
Projected  Host— Host  traffic  estimates  were  known  at 
the  start  to  be  either  unreliable  or  wrong.  Therefore, 
the  network  was  designed  under  the  assumption  of 
equal  traffic  between  all  pairs  of  nodes.  (Additional 
superimposed  traffic  was  sometimes  included  for  those 
nodes  with  expectation  of  higher  traffic  requirements.) 
The  topological  structure  was  determined  with  the  aid 
of  specially  developed  heuristic  programs  to  achieve  a 
low  cost,  reliable  network  with  a  high  throughput  and 
a  general  insensitivity  to  the  exact  traffic  distribution. 
Currently,  only  50  kilobit/second  circuits  are  being 
used  in  the  ARPANET.  This  speed  line  was  chosen  to 
allow  rapid  transmission  of  short  messages  for  inter¬ 
active  processing  (e.g.,  less  than  0.2  seconds  average 
packet  delay)  as  well  as  to  achieve  high  throughput 
(e.g ,  at  least  50  kilobits/second)  for  transmission  of 
long  messages.  For  reliability,  the  network  was  con¬ 
strained  to  have  at  least  two  independent  paths  between 
cacn  pair  o!  fMf  a. 


The  topological  design  problem  requires  consideration 
of  the  following  two  questions: 

(1)  Starting  with  a  given  state  of  the  network 
topology,  what  circuit  modifications  art'  required 
to  add  or  delete  a  set  of  1  lM  Ps? 

(2)  Starting  with  a  given  state  of  network  topology, 
when  and  where  should  circuits  be  added  or 
deleted  to  account  for  long  term  changes  in 
network  traffic? 

If  the  locations  of  all  network  nodes  are  known  in 
advance,  it  is  clearly  most  efficient  to  design  the 
topological  structure  as  a  single  global  effort.  However, 
in  the  ARPANET,  as  in  most  actual  net  wot  ks,  the 
initial  designation  of  node  locations  is  modified  on 
numerous  occasions.  On  each  such  occasion,  the 
topology  can  be  completely  reoptimized  to  determine  a 
new  set  of  circuit  locations. 

In  practice,  there  is  a  long  lead  time  between  the 
ordering  and  the  delivery  of  a  circuit,  and  major  topo¬ 
logical  modifications  cannot  be  made  without  sub¬ 
stantial  difficulty.  It  is  therefore  prudent  to  add  or 
delete  nodes  with  as  little  disturbance  as  possible  to 
the  basic  network  structure  consistent  with  overall 
economical  operation.  Figure  1  shows  the  evolution  of 
the  ARPANET  from  the  basic  four  IMP  design  in  Hit')!! 
to  the  presently  planned  27  IMP  version.  Inspection  of 
the  24  and  27  IMP  hetwurk  diaigna  i>  V  ..Is  ..  f;  w 
substantial  changes  in  topology  that  take  advantage  of 
tho  new  nodes  being  added.  Surprisingly  enough,  a 
complete  "reoptimization”  of  the  27  IMP  topology 
yields  a  network  only  slightly  less  expensive  (about 
1  percent)  than  the  present  network  design.28 

Network  models 

The  development  of  an  accurate  mathematical  model 
for  the  evaluation  of  time  delay  in  computer  networks  is 
among  the  more  difficult  of  the  topics  discussed  in  this 
paper.  On  the  one  hand,  the  model  must  properly 
reflect  the  relevant  features  of  the-  network  structure 
and  operation,  including  practical  constraints.  On  the 
other  hand,  the  model  must  result  in  a  mathematical 
formulation  which  is  tractable  and  from  which  mean¬ 
ingful  results  can  be  extracted  However,  the  two 
requirements  are  often  incompatible  and  we  search  for 
an  acceptable  compromise;  between  these  two  extremi  s. 

The  major  modeling  effort  thus  far  has  been  the  studs 
of  the  behavior  of  networks  of  queues.*'  This  emphasis 
is  logical  since  in  message  switched  systems,  messages 
experience  queueing  delays  as  they  pass  front  node  to 
oo*  »4gIlUIrjet  performance  measure  is  the 
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Figure  1— The  evolution  of  the  ARPANET 


speed  at  which  messages  can  be  delivered.  The  queueing 
models  were  developed  at  a  time  when  there  were  no 
operational  networks  available  for  experimentation  and 
model  validation,  and  simulation  was  the  only  tool 
capable  of  testing  their  validity.  The  models,  which  at 
all  times  were  recognized  to  be  idealized  statements 
about  the  real  network,  were  nonetheless  crucial  to  the 
ARP ANET  topological  design  effort  since  they  afforded 
the  only  known  way  to  quantitatively  predict  the 
properties  of  different  routing  schemes  and  topological 
structures.  The  models  have  been  subsequently  demon¬ 
strated  to  be  very  accurate  predictors  of  network 
throughput  and  indispensable  in  providing  analytical 
insight  into  the  network’s  behavior. 

The  key  to  the  successful  development  of  tractable 
models  has  been  to  factor  the  problem  into  a  set  of 
simpler  queueing  problems.  There  are  also  heuristic 
design  procedures  that  one  can  use  in  this  case.  These 
procedures  seem  to  work  quite  well  and  arc  described 
later  in  the  paper.  However,  if  one  specializes  the 
problem  and  removes  some  of  the  real  constraints, 
theory  and  analysis  become  useful  to  provide  under¬ 
standing,  intuition  and  design  guidelines  for  the  original 
constrained  problem.  This  approach  uncovers  global 
properties  of  network  behavior,  which  provide  keys  to 


good  heuristic  design  procedures  and  ideal  performance 
bounds. 


DESIGN  TECHNIQUES 

In  this  section  we  describe  the  approaches  taken  to 
the  design  problems  introduced  in  the  previous  section. 
We  first  summarize  the  important  properties  of  the 
ARPANET  design: 

(1)  A  communications  cost  of  less  than  30  cents  per 
thousand  packets  (approximately  a  megabit). 

(2)  Average  packet  delays  under  0.2  seconds  through 
the  net . 

(3)  Capacity  for  expansion  to  64  IMPs  without 
major  hardware  or  software  redesign. 

(4)  Average  total  throughput  capability  of  10-15 
kilobits/second  for  all  Hosts  at  an  IMP. 

(5)  Peak  throughput  capability  of  85  kilobits/second 
per  pair  of  IMPs  in  an  otherwise  unloaded 
network. 

(6)  Transparent  communications  with  maximum 
message  size  of  approximately  8000  bits  and 
error  rates  of  one  bit  in  10”  or  less, 
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(T)  Approximately  9S  percent  availability  of  any 
IMP  and  close  to  100  |x*rcei.t  availability  of  all 
(•Iterating  I.MPs  from  any  operable  fMP. 

The  relationships  bt'livcni  the  various  design  efforts 
art'  illustrated  by  these  properties.  The  to|Kilogical 
design  provides  for  both  a  desir  si  average  tiiroughpnt 
and  for  two  or  more  paths  to  be  fully  used  for  com¬ 
munication  between  any  pair  of  Hosts.  The  operating 
procedure  should  allow  any  pair  of  Hosts  to  achieve 
those  obj.  ctives.  The  availability  of  I.MPs  to  com¬ 
municate  reflects  both  the  fact  that  I.MPs  are  down 
about  2  percent  of  the  time  and  that  the  to|>olngy  is 
selected  so  that  circuit  failures  contribute  little  addi¬ 
tional  to  the  total  system  downtime. 


/ MP  design 

The  IMP  design  consists  of  two  closely  coupled  but 
nonetheless  separable  pieces— the  physical  hardware 
specification  (based  on  timing  and  reliability  considera¬ 
tions  and  the  operating  procedure)  and  the  design  and 
implementation  of  the  operating  procedure  using  the 
specified  IMP  hardware.  The  IMP  originally  developed 
for  the  ARPANKT  contains  a  16-bit  one  microsecond 
computer  that  can  handle  a  total  of  about  megabits/ 
second  of  “useful”  information  on  a  total  of  approxi¬ 
mately  one  megabit /second  of  circuit  capacity  (c.g., 
twenty  50  kilobit/second  circuits).  Hardware  is  likely 
to  change  as  a  function  of  the  required  IMP  capacity 
but  an  operating  procedure  that  operates  well  at  one 
IMP  capacity  is  likely  to  be  transferable  to  machines 
that  provide  different  capacity.  However,  as  a  network 
grows  in  size  and  utilization,  a  mere  comprehensive 
operating  procedure  that  takes  account  of  known 
structural  properties,  such  as  a  hierarchical  topology, 
is  appropriate. 

Four  primary  areas  of  IMP  design,  namely  message 
handling  and  buffering,  error  control,  flow  control,  and 
routing  arc  discussed  in  this  section.  The  IM  P  provides 
buffering  to  handle  messag.  j  for  its  Host  and  packets 
for  other  IMPs  Error  control  is  required  to  provide 
reliable  communication  of  Host  messages  in  the 
presence  of  noisy  communication  circuits.  The  design 
of  the  operating  procedure  should  allow  high  through¬ 
put  in  the  net  under  heavy  traffic  loads.  Two  potential 
obstacles  to  achieving  this  objective  ore:  (1)  The  net 
can  become  congested  and  cause  the  throughput  to 
decrease  with  increasing  load,  and  (2)  The  routing 
procedure  may  be  unable  to  i.Iways  adapt  sufficiently 
fast  to  the  rapid  movement  of  packets  to  insure  efficient 
routing.  A  flow  control  and  routing  procedure  is 
needed  that  can  efficiently  meet  this  requirement. 


.Message  handling  and  buffering 

In  the  ARPANET,  the  maximum  message  siz*  was 
constrained  to  lie  approximately  S(XX)  bits.  A  pair  of 
Hosts  will  typically  communicate  over  the  net  via  a 
sequence  of  transmitted  messages.  To  obtain  delaxs  of 
a  few  tenths  of  a  second  for  such  messages  and  to  lower 
the  required  IMP  buffer  storage,  the  IMP  program 
partitions  each  message  into  one  or  more  packets  each 
containing  at  most  approximately  1000  bits.  Each 
packet  of  a  message  is  transmitted  independently  to 
the  destination  where  tlx*  message  is  reassembled  by 
the  IMP  before  shipment  to  that  destination  Host. 
Alternately,  the  Hosts  could  assume  the  responsibility 
for  reassembling  messages  For  an  asynchronous  IMP- 
Host  channel,  this  marginally  simplifies  the  IMP- 
task.  However,  if  eirry  IMP-IIost  channel  were  >\  n- 
chronous,  and  the  Host  providisl  the  reassembly.  >!.. 
IMP  task  can  b<  further  simplified  In  this  latter 
"IMP-like"  software  would  have  to  be  provid'd  in 
each  Host. 

The  method  of  handling  buffers  should  be  simple  to 
allow  for  fast  processing  and  a  small  amount  of  program. 
The  number  of  buffers  should  he  sufficient  to  sten 
enough  packets  for  the  circuits  to  be  used  to  capacity; 
the  size  of  the  buffers  may  be  intuitively  select. d  with 
the  aid  of  sim*de  analytical  techniques.  For  example, 
fixed  buffer  sizes  are  useful  in  the  IMP  for  simplicity 
of  design  and  speed  of  operation,  but  inefficient  utiliza¬ 
tion  can  arise  because  of  variable  length  packets.  If 
each  buffer  contains  A  words  of  overhead  ami  provides 
space  for  M  words  of  text,  and  if  message  sizes  are 
uniformly  distributed  between  1  nnd  I,,  it  can  be 
shown”  that  the  choice  of  M  that  minimizes  the  e\- 
liected  storage  is  approximately  y'Al.  In  practice.  M 
is  chosen  to  be  somewhat  smaller  on  the  assumption 
that  most  traffic  will  be  short  and  that  the  animnv  of 
overhead  can  be  as  much  as,  say,  25  percent  of  buffer 
storage. 


Error  control 

The  IMPs  must  assume  the  res|K>nnihility  for  pro¬ 
viding  error  control.  There  are  four  possibilities  to 
consider: 

(1)  Messages  are  delivered  to  their  destination  out 
of  order. 

(2)  Duplicate  messages  are  delivered  to  the 
destination. 

(3)  Messages  are  delivered  with  errors 

(4)  Messages  are  not  delivered. 
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The  task  of  proper  sequencing  of  messages  for 
delivery  to  the  destination  Host  actually  falls  in  the 
province  ,,f  both  error  control  and  flow  control.  If  at 
most  one  message  at  a  time  is  allowed  in  the  net  bet  ween 
a  pair  of  Hosts,  proper  sequencing  occurs  naturally.  A 
duplicate  packet  will  arrive  at  the  destination  INI  I’ 
after  an  acknowledgment  has  boon  missed,  thus  causing 
a  successfully  received  packet  to  be  retransmitted.  The 
IMPs  can  handle  the  first  two  conditioas  by  assigning 
a  sequence  number  to  each  packet  as  it  enters  the 
network  and  processing  the  sequence  number  at  the 
destination  IMP.  A  Host  that  performs  reassembly  can 
also  assign  and  process  sequence  numbers  and  check 
for  duplicate  packets.  For  many  applications,  the  order 
of  delivery  to  the  destination  is  immaterial.  For  priority 
messages,  however,  it  is  typically  the  case  that  fast 
delivery  requires  a  perturbation  to  the  sequence. 

Errors  are  primarily  caused  by  noise  on  the  com¬ 
munication  circuits  and  are  handled  most  simply  by 
error  detection  and  retransmission  between  each  pair  of 
IMPs  along  the  transmission  path.  This  technique 
requires  extra  storage  in  the  IMP  if  either  circuit 
speeds  or  circuit  lengths  substantially  increase.  Failures 
in  detecting  errors  can  be  made  to  occur  on  the  order  of 
years  to  centuries  apart  with  little  extra  overhead 
(20-30  parity  bits  per  packet  with  the  50  kilobit /second 
circuits  in  the  ARPANET).  Standard  cyclic  error 
detection  codes  have  been  usefully  applied  hi  re. 

A  reliable  system  design  insures  that  each  trans¬ 
mitted  message  is  accurately  delivered  to  its  intended 
destination.  The  occasional  time  when  an  IMP  fails  and 
destroys  a  useful  in-transit  message  is  likely  to  occur 
far  less  often  than  a  similar  failure  in  the  Hosts  and  has 
proven  to  be  unimportant  in  practice,  as  are  errors  due 
to  IMP  memory  failures.  A  simple  end  to  end  retrans¬ 
mission  strategy  will  protect  against  these  situations, 
if  the  practical  need  should  arise.  However,  the  IMPs 
arc  designed  so  that  they  can  be  removed  from  the 
network  without  destroying  their  internally  stored 
packets. 


Flow  control 

A  network  in  which  packets  may  freely  enter  and 
leave  can  become  congested  or  logically  deadlocked  and 
cause  the  movement  of  traffic  to  halt.* 17  How  control 
techniques  arc  required  to  prevent  these  conditions 
from  occurring.  The  provision  of  extra  buffer  storage 
will  mitigate  against  congestion  and  deadlocks,  but 
cannot  in  general  prevent  them. 

The  sustained  failure  of  a  destination  Host  to  accept 
packets  from  its  IMP  at  the  rate  of  arrival  will  cause  the 
net  to  fill  up  and  become  congested.  Two  kinds  of 


logical  deadlocks,  known  as  reassembly  lockup  and 
store-n nil-forward  lin  kup  may  also  occur.  In  reassembly 
lockup,  the  remaining  packets  of  partially  r.  assembled 
messages  are  blocked  from  reaching  the  destination 
IMP  (thus  preventing  the  messagi  from  being  com¬ 
pleted  and  the  reassembly  space  fr.-ed)  by  other 
packets  in  the  net  that  are  waiting  for  reassembly  spat 
at  that  destination  to  become  free.  In  a  store-and- 
forwaril  lockup,  the  destination  has  room  to  accept 
arriving  packets,  hut  the  packets  interfere  will,  each 
other  by  tying  up  buffers  in  transit  in  such  a  way  that 
none  of  the  packets  are  able  to  reach  the  destination.*’ 
These  phenomena  have  only  been  made  to  occur  during 
very  carefully  arranged  testing  of  the  ARPANET  and 
by  simulation.0 

'  In  the  original  ARPANET  design,  the  use  of  soft¬ 
ware  links  and  KFN.MS  protected  against  congestion 
by  a  single  link  or  a  small  set  of  links.  However,  the 
combined  traffic  on  a  large  number  of  links  could  still 
produce  congestion.  Although  this  strategy  did  not 
protect  against  lockup,  the  method  has  provided  ample 
protection  for  the  levels  of  traffic  encountered  b\  the 
net  to  date. 

A  particularly  simple  flow  control  algorithm  that 
nugments  the  original  IMP  design  to  prevent  eoiigest  ioii 
and  lockup  is  also  described  in  Reference  17.  '1  Ins 
scheme  includes  a  mechanism  whereby  packets  may  be 
discarded  from  the  net  at  the  destination  IMP  when 
congestion  is  about  to  occur,  with  a  copy-  "f  each  dis¬ 
carded  packet  to  be  retransmitted  a  short  time  later  by 
the  originating  Host’s  IMP.  Rather  than  expericnc. 
excessive  delays  within  the  net  as  traffic  levels  are 
increased,  the  traffic  is  queued  outside  the  net  so  that 
the  transit  time  delays  internal  to  the  net  continue  to 
remain  small.  This  strategy  prevents  the  insertion  of 
more  traffic  into  the  net  than  it  can  handle. 

It  is  important  to  note  the  dual  requirement  for  small 
delays  for  interactive  traffic  and  high  bandwidth  lor  tin- 
fast  transfer  of  files.  To  allow  high  bandwidth  between 
a  pair  of  Hosts,  the  net  must  be  able  to  acc  |U  a  ‘toady 
flow  of  packets  from  one  Host  and  at  the  >um  ' un¬ 
able  to  rapidly  quench  the  flow  at  the  entrant*  t-  h* 
source  IMP  in  the  event  of  imminent  congestion  at  tin* 
destination.  This  usually  requires  that  a  separate 
provision  he  made  in  the  algorithm  to  protect  short 
interactive  messages  from  experiencing  unnecessarily 
high  delays. 

Routing 

Network  routing  strategies  for  distributed  networks 
require  routing  decisions  to  be  made  with  only  in¬ 
formation  available  to  an  IMP  and  the  IM1  must 
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execute  those  decisions  to  cffi'Ct  the  routing.""  A 
simple  example  of  such  a  strategy  is  to  have  each  IMP 
handling  a  packet  indejs-ndentlv  route  it  along  its 
current  estimate  of  th<*  shortest  path  to  the  destination. 

For  many  applications,  it  suffices  to  deal  with  an 
idealized  routing  strategy  which  may  not  simulate  the 
IMP  routing  functions  in  detail  or  which  uses  informa¬ 
tion  not  available  to  the  IMP  The  general  properties 
of  both  strategies  are  usually  similar,  differing  mainly 
in  certain  implementation  details  such  as  the  avail¬ 
ability  of  buffers  or  the  constraint  of  counters  and  the 
need  for  the  routing  to  quickly  adapt  to  changes  in 
IMP  and  circuit  status. 

The  IMPs  perform  the  routing  computations  using 
information  received  from  other  IMPs  and  local 
information  such  as  the  alive/dead  state  of  its  circuits. 
In  the  normal  case  of  time  varying  loads,  local  informa¬ 
tion  alone,  such  as  the  length  of  internal  queues,  is 
insufficient  to  provide  an  efficient  routing  strategy 
without  assistance  from  the  neighboring  IMPs.  It  is 
possible  to  obtain  sufficient  information  from  the 
neighbors  to  provide  efficient  routing,  with  a  small 
amount  of  computation  needed  per  IMP  and  without 
each  IMP  requiring  a  topological  map  of  the  network. 
In  certain  applications  where  traffic  patterns  exhibit 
regularity,  the  use  of  a  central  controller  might  In- 
preferable.  However,  for  most  applications  which 
involve  dynamically  varying  traffic  flow,  it  appears 
that  a  central  controller  cannot  lie  used  more  effectively 
than  the  IMPs  to  update  routing  tables  if  such  a 
controller  is  coastrained  to  derive  its  information  via 
the  net.  It  is  also  a  less  reliable  approach  to  routing 
than  to  distribute  the  routing  decisions  among  the 
IMPs. 

The  routing  information  cannot  be  propagated  about 
the  net  in  sufficient  time  to  accurately  characterize  the 
instantaneous  traffic  flow.  An  efficient  algorithm,  there¬ 
fore,  should  not  focus  on  the  movement  of  individual 
packets,  but  rather  use  topological  or  statistical  in¬ 
formation  in  the  selection  of  routes.  For  example,  by 
using  an  averaging  procedure,  the  flow  of  traffic  can  be 
made  to  build  up  smoothly.  This  allows  the  routing 
algorithm  ample  time  to  adjust  its  tables  in  each  IMP 
in  ail vance  of  the  build-up  of  traffic. 

The  scheme  originally  used  in  the  AH  PA  network 
hail  each  IMP  select  one  output  line  per  destination 
onto  which  to  route  packets.  The  line  was  chosen  to  be 
the  one  with  minimum  estimated  time  delay  to  the 
destination,  The  seb'ction  was  updated  every  half 
second  using  minimum  time  estimates  from  the  neigh¬ 
boring  IMPs  and  internal  estimate.,  of  the  delay  to  each 
of  the  neighbors  Even  though  the  routing  algorithm 
only  selects  one  line  at  a  time  per  destination,  two 
output  lines  will  be  used  if  a  queue  of  packets  waiting 


transmission  on  one  line  builds  up  before  the  routing 
update  occurs  and  another  line  is  chosen.  Modifications 
to  the  scheme  which  allow  several  lines  per  destination 
to  In*  used  in  an  update  interval  (during  which  the 
routing  is  not  changed)  are  possible  using  two  or  more 
time  delay  estimates  to  select  the  paths. 

In  practice,  this  approach  has  worked  quite  effectively 
with  the  moderate  levels  of  traffic  experienced  in  the 
net.  For  heavy  traffic  flow,  this  strategy  may  be 
inefficient,  since  the  routing  information  is  based  on 
the  length  of  queues,  which  we  have  seen  can  change 
much  faster  than  the  information  about  the  change  can 
be  distributed.  Fortunately,  this  information  is  still 
usable,  although  it  can  bo  substantially  out  of  date  and 
will  not,  in  general,  be  helpful  in  making  efficient 
routing  decisions  in  the  heavy  traffic  case. 

A  more  intricate  scheme,  recently  developed  by 
BBX,  allows  multiple  paths  to  be  efficiently  used  ev«  n 
during  heavy  traffic."  Preliminary  simulation  stu  b  - 
indicate  that  it  can  be  tailored  to  provide  efficient 
routing  in  a  network  with  a  variety  of  heavy  traffic 
conditions.  This  method  separates  the  problem  of 
defiiiing  routes  onto  which  packets  may  he  routed  from 
the  problem  of  selecting  n  route  when  a  particular 
packet  must  hi*  routed.  By  this  technique,  it  is  possible 
to  send  packets  down  a  path  with  the  fewest  IMPs  and 
excess  capacity,  or  when  that  path  is  filled,  the  one  with 
the  next  fewest  IMPs  and  excess  capacity,  etc. 

A  similar  approach  to  routing  was  independently 
derived  by  X AC  using  nn  idealized  method  that  did  not 
require  the  IMPs  to  participate  in  the  routing  decisions. 
Another  approach  using  a  flow  deviation  technique  has 
recently  been  under  study  at  UCLA."  The  intricacies  of 
the  exact  approach  lead  to  a  metering  procedure  that 
allows  the  overall  network  flow  to  be  changeil  slow  I y  for 
stability  and  to  perturb  existing  flow  patterns  to  obtain 
an  increased  flow.  These  approaches  all  possess,  in 
common,  essential  ingmlicnts  of  u  desirable  routing 
strategy. 

T opological  considerations 

An  efficient  topological  design  provides  a  high 
throughput  for  a  given  cost  Although  many  measures 
of  throughput  are  possible,  a  convenient  one  is  the 
average  amount  of  traffic  that  a  single  IMP  can  send 
oito  the  network  when  all  other  IMPs  are  transmitting 
according  to  a  specified  traffic  pattern.  Often,  it  is 
assumed  that  all  other  IMPs  are  behaving  identically 
and  each  IMP  is  sending  equal  amounts  of  traffic  to 
each  other  IMP.  The  constraints  on  the  topological 
design  are  the  available*  common  carrier  circuits,  the 
target  cost  or  throughput,  the  desired  reliability,  and 
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TABLE  1—23  Node  28  Link  Al{  PA 


Reliability  computations 


Number  of 
Circuits 
Failed 

Number  of 
Combinations 
to  be  Examined 

Number  of 
Cutsets 

28 
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27 

28 

28 
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21 

1184040 
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8 

3108108 
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7 
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6 

376740 

349618 

O 

98280 
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4 

20473 

=■9832 

3 

3276 

827 

2 

378 

30 

1 

28 

0 

the  cost  of  computation  required  to  perform  the 
topological  design. 

Since,  there  was  no  clear  specification  of  the  amount 
of  traffic  that  the  network  would  have  to  accommodate 
initially,  it  was  first  constructed  with  enough  excess 
capacity  to  accommodate  any  reasonable  traffic  require¬ 
ments.  Then  as  new  IMPs  were  added  to  the  system, 
the  capacity  was  and  is  still  being  systematically 
reduced  until  the  traffic  level  occupies  a  substantial 
fraction  of  the  network’s  total  capacity.  At  this  point, 
the  net’s  capacity  will  be  increased  to  maintain  the 
desired  percentage  of  loading.  At  the  initial  stages  of 
network  design,  the  “two-connected"  reliability  con¬ 
straint  essentially  determined  a  minimum  value  of 
maximum  throughput.  This  constraint  forces  the. 
average  throughput  to  be  in  the  range  10-15  kilobits  per 
second  per  IMP,  when  50  kilobit/sec  circuits  are  used 
throughout  the  network,  since  two  communication 
paths  between  every  pair  of  IMPs  are  needed  Alterna¬ 
tively,  if  this  level  of  throughput  is  required,  then  the 
reliability  specification  of  “two-connectivity"  can  be 
obtained  without  additional  cost 


A  simple  and  natural  characterization  of  network 
reliability  is  the  ability  of  the  network  to  sustain 
communication  between  all  operable  pairs  of  IMPs.  For 
design  purposes,  the  requirement  of  two  independent 
paths  between  nodes  insures  that  at  least  two  IMPs 
and/or  circuits  must  fail  before  any  pair  of  operable 
IMPs  cannot  communicate.  This  criterion  is  independent 
of  the  properties  of  the  IMPs  and  circuits,  does  not  take 
into  account  the  “degree”  of  disruption  that  may  occur 
and  hence,  docs  not  reflect  the  actual  availability  of 
resources  in  the  network.  A  more  meaningful  measure 
is  the  average  fraction  of  IMP  pairs  that  canm  t  com¬ 
municate  because  of  IMP  and  circuit  failures.  Thi- 
calculation  requires  knowledge  of  the  IMP  and  circuit 
failure  rates,  and  could  not  be  performed  until  enough 
operating  data  was  gathered  to  make  valid  prediction*. 

To  calculate  network  reliability,  we  must  cmed'h  r 
elementary  network  structures  known  as  cutsets.  A 


Figure  2 — Network  availability  vs.  IMP  unit  circuit  reliability 
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cutset  is  a  set  of  circuits  and.  or  I.MPs  whose  removal 
from  the  network  breaks  all  communication  paths 
between  at  least  two  operable  I  Mi’s.  To  calculate 
reliability,  it  is  often  the  case  that  all  cutsets  must  be 
either  enumerated  or  estimated.  As  an  example,  in  a 
23  IMP,  2S  circuit  ARl’A  Network  design  similar  to 
the  one  shown  in  Figure  1(d),  there  are  over  twenty 
million  ways  of  deleting  only  circuits  so  that  the 
remaining  network  has  at  least  one  operable  pair  of 

1. MPs  with  no  intact  communication  paths.  Table  1 
indicates  the  numbers  of  cutsets  in  the  23  IMP  network 
as  a  function  of  the  number  of  circuits  they  contain. 

A  combination  of  analysis  and  simulation  can  be 
used  to  compute  the  average  fraction  of  non-com- 
nnmicating  IMP  pairs.  Detailed  descriptions  of  the 
analysis  methods  are  given  in  Reference  44  while  their 
application  to  the  analysis  of  the  ARPANET  is  dis¬ 
cussed  in  Reference  43.  The  results  of  an  analysis  of 
the  23  IMP  version  of  the  network  are  shown  in  Figure 

2.  The  curve  marked  A  shows  the  results  under  the 
assumption  that  I.MPs  do  not  fail,  while  the  curve 
marked  B  shows  the  case  where  circuits  do  not  fail. 
The  curve  marked  C  assumes  that  both  IMl’s  and 
circuits  fail  with  equal  probability.  In  actual  operation, 
the  average  failure  probability  of  both  IMPs  and 
circuits  is  about  0.02.  For  this  value,  it  can  be  seen  that 
the  effect  of  circuit  failures  is  far  less  significant  than 
the  effect  of  IMP  failures.  If  an  IMP  fails  in  a  network 
with  n  IMPs,  at  least  a  — 1  other  I.MPs  cannot  com¬ 
municate  with  it.  Thus,  good  network  design  cannot 
improve  upon  the  effect  directly  due  to  IMP  failures, 
which  in  the  ARPANET  is  the  major  factor  affecting 
the  reliability  of  the  communications.  Further,  more 
intricate  reliability  analyses  which  consider  the  loss  of 
throughput  capacity  because  of  circuit  failures  have 
also  been  performed  and  these  losses  have  been  shown 
to  be  negligible. M  Finally,  unequal  failure  rates  due  to 
differences  in  line  lengths  have  been  shown  to  have 
only  minor  effects  on  the  analysis  and  can  usually  be 
neglected.” 

Topological  optimization 

During  the  computer  optimization  process,  the 
reliability  of  the  topology  is  assumed  to  be  acceptable  if 
the  network  is  at  least  two-connected.  The  object  of 
the  optimization  is  to  decrease  the  ratio  of  cost  to 
throughput  subject  to  an  overall  cost  limitation.  This 
technique  employs  a  sophisticated  network  optimization 
program  that  utilizes  circuit  exchange  heuristics, 
routing  and  flow  analysis  algorithms,  to  generate  low- 
cost  designs.  In  addition,  two  time  delay  models  were 
initially  used  to  (1)  calculate  the  throughput  corre¬ 


sponding  to  an  average  time  delay  of  0.2  seconds, 
(2)  estimate  the  packet  rejection  rate  due  to  all  buffers 
filling  at  an  IMP.  As  experience  with  these  models 
grew,  the  packet  rejection  rate  was  found  to  be  negligible 
and  the  computation  discontinued.  The  delay  computa¬ 
tion  (Equation  (7)  in  a  later  section)  was  subsequently 
first  replaced  by  a  heuristic  calculation  to  speed  the 
computation  and  later  eliminated  after  it  was  found 
that  time  delays  could  be  guaranteed  to  be  acceptably 
low  by  preventing  cutsets  from  being  saturated.  This 
“threshold"  behavior  is  discussed  further  in  the  next 
section. 

The  basic  method  of  optimization  was  described  in 
Reference  12  while  extensions  to  the  design  of  large 
networks  are  discussed  in  Reference  9.  The  method 
operates  by  initially  generating,  either  manually  or  by 
computer,  a  “starting  network"  that  satisfies  the  overall 
network  constraints  but  is  not,  in  general,  a  low  c-t 
network.  The  computer  then  iteratively  modifies  the 
starting  network  in  simple  steps  until  a  lower  cost 
network  is  found  that  satisfies  the  constraints  or  the 
process  is  terminated.  The  process  is  repeated  until  no 
further  improvements  can  be  found.  Using  a  different 
starting  network  can  result  in  a  different  solution. 
However,  by  incorporating  sensible  heuristics  and  by 
using  a  variety  of  carefully  chosen  starting  networks  and 
some  degree  of  man-machine  interaction,  "excellent” 
final  networks  usually  result.  Experience  has  shown 
that  there  are  a  wide  variety  of  such  networks  with 
different  topological  structures  but  similar  cost  and 
performance. 

The  key  to  this  design  effort  is  the  heuristic  procedure 
by  which  the  iterative  network  modifications  are  made. 
The  method  used  in  the  ARPANET  design  involves  the 
removal  and  addition  of  one  or  two  circuits  at  a  time. 
Many  methods  have  been  employed,  at  various  times, 
to  identify  the  appropriate  circuits  for  potential  addi¬ 
tion  or  deletion.  For  example,  to  delete  uneconomical 
circuits  a  straightforward  procedure  simply  deletes 
single  circuits  in  numerical  order,  reroutes  traffic  and 
reevaluates  cost  until  a  decrease  in  cost  per  megabit  is 
found.  At  this  point,  the  deletion  is  made  permanent 
and  the  process  begins  again.  A  somewhat  more 
sophisticated  procedure  deletes  circuits  in  order  of 
increasing  utilization,  while  a  more  complex  method 
attempts  to  evaluate  the  effect  of  the  removal  of  any 
circuit  before  any  deletion  is  attempted.  The  cireuit 
with  the  greatest  likelihood  of  an  improvement  is  then 
considered  for  removal  and  so  on 

There  are  a  huge  number  of  reasonable  heuristics  for 
circuit  exchanges.  After  a  great  deal  of  experimentation 
with  many  of  these,  it  appears  that  the  choiei  of  a 
particular  heuristic  is  not  critical.  Instead,  tlm  spei  d 
and  efficiency  with  which  potential  exchanges  can  be 
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investigated  appears  to  he  the  limiting  factor  affecting 
the  quality  of  the  final  design.  Finally,  as  the  size  of 
the  network  increases,  the  greater  the  cost  becomes  to 
perform  any  circuit  exchange  optimization.  Decom¬ 
position  of  the  network  design  into  regions  becomes 
necessary  and  additional  heuristics  are  needed  to 
determine  effective  decompositions.  It  presently  appears 
that  these  methods  can  be  used  to  design  relatively 
efficient  networks  with  a  few  hundred  IMPs  while 
substantially  new  procedures  will  be  necessary  for 
networks  of  greater  size. 

The  topological  design  requires  a  routing  algorithm 
to  evaluate  the  throughput  capability  of  any  given 
network.  Its  properties  must  reflect  those  of  an  im- 
plementable  routing  algorithm,  for  example,  within 
the  ARPANET.  Although  the  routing  problem  can  be 
formulated  as  a  “multicommodity  flow  problem”10  and 
solved  by  linear  programming  for  networks  with  20-30 
IMPs,8  faster  techniques  are  needed  when  the  routing 
algorithm  is  incorporated  in  a  design  procedure.  The 
design  procedure  for  the  ARPA  Network  topology 
iteratively  analyzes  thousands  of  networks.  To  satisfy 
the  requirements  for  speed,  an  algorithm  which 
selects  the  least  utilized  path  with  the  minimum  number 
of  IMPs  was  initially  used.1*  This  algorithm  was  later 
replaced  by  one  which  sends  as  much  traffic  as  possible 
along  such  paths  until  one  or  more  circuits  approach  a 
few  percent  of  full  utilization.”  These  highly  utilized 
circuits  are  then  no  longer  allowed  to  carry  additional 
flow.  Instead,  now  paths  with  excess  capacity  and 
possibly  more  intermediate  nodes  are  found.  The 
procedure  continues  until  some  cutset  contains  only 
nearly  fully  utilized  circuits.  At  this  point  no  additional 
flow  can  be  sent.  For  design  purposes,  this  algorithm  is 
a  highly  satisfactory  replacement  for  the  more  com¬ 
plicated  multi-commodity  approach.  Using  the  al¬ 
gorithm,  it  has  been  shown  that  the  throughput  capa¬ 
bilities  of  the  ARPA  Network  are  substantially 
insensitive  to  the  distribution  of  traffic  and  depend 
mainly  only  on  the  total  traffic  flow  within  the  network.* 


In  Reference  22,  it  was  modified  to  describe  the  be¬ 
havior  of  ARPA-like  computer  networks  while  in 
Reference  23  it  was  refined  further  to  apply  direct  1\  to 
the  ARPANET. 


The  single  server  model 


Queueing  theory50  provides  an  effective  set  of  ana¬ 
lytical  tools  for  studying  packet  delay.  Much  of  this 
theory  considers  systems  in  which  messages  place 
demantis  for  transmission  (service)  upon  a  single 
communication  channel  (tht1  single  server).  These 
systems  tire  characterized  by  .4(r),  the  distribution  ot 
interarrival  times  between  demands  and  the 

distribution  of  service  times.  When  the  average  demand 
for  service  is  less  than  the  capacity  of  the  channel,  the 
system  is  said  to  be  stable. 

When  A(t)  is  exponential  (i.e.,  Poisson  arrivals), 
and  messages  are  transmitted  on  a  first -come  first-served 
basis,  the  average  time  T  in  the  stable  system  is 


7  = 


2(1 -p) 
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where  X  is  the  average  arrival  rate  of  messages,  t  ami  t- 
are  the  first  and  second  moments  of  /{(()  respectively, 
and  p=X(’<  1.  If  the  service  time  is  also  exponential, 

T=  — —  (2) 

1  -p 


When  A  (r)  and  li(t)  are  arbitrary  distributions,  the 
situation  becomes  complex  and  only  weak  results  are 
available.  For  example,  no  expression  is  available  for  V: 
however  the  following  upper  bound  yields  an  excellent 
approximation18  as  p— ♦! : 
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where  o<?  and  as1  tin*  the  variance  of  the  intorurrival 
time  and  service  time  distributions,  respectively. 


Analytic  models  of  network  performance 

The  effort  to  determine  analytic  models  of  system 
performance  has  proceeded  in  two  phases:  (1)  the  pre¬ 
diction  of  average  time  delay  encountered  by  a  message* 
as  it  passes  through  the  network,  and  (2)  the  use  of 
these  queueing  models  to  calculate  optimum  channel 
capacity  assignments  for  minimum  puss'  hie  delay.  The 
model  used  as  a  standard  for  the  average  message  delay 
was  first  described  in  Reference  21  where  it  served  to 
predict  delays  in  stochastic  communication  networks. 


Networks  of  queues 

Multiple  channels  in  a  network  environment  giv 
rise  to  queueing  problems  that  are  far  more  difficult  to 
solve  than  single  server  systems.  For  example,  the 
variability  in  the  choice  of  source  and  destination  for  a 
message  is  a  network  phenomenon  which  contributes  to 
delay.  A  principal  analytical  difficulty  results  from  the 
fact  that  flows  throughout  the  network  tire  correlat'd. 
The  basic  approach  to  solving  these  stochastic  network 
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problems  is  to  decompose  them  into  analyzable  single- 
server  problems  which  reflect  the  original  network 
structure  and  traffic  flow. 

Early  studies  of  queueing  networks  indicated  that 
such  a  decomposition  was  possible however,  those 
results  do  not  carry  over  to  message  switched  computer 
networks  due  to  the  correlation  of  traffic  flows  In 
Reference  21  it  was  shown  for  a  wide  variety  of  com¬ 
munication  nets  that  this  correlation  could  be  removed 
by  considering  the  length  of  a  given  packet  to  be  an 
independent  random  variable  as  it  passes  from  node’ to 
node.  Although  this  "independence”  assumption  is  not 
physically  realistic,  it  results  in  a  mathematically 
tract  able  model  which  does  not  seem  to  affect  the 
accuracy  of  the  predicted  time  delays.  As  the  size  and 
connectivity  of  the  network  increases,  the  assumption 
becomes  increasingly  more  realistic.  With  this  assump¬ 
tion,  a  successful  decomposition  which  permits  a 
channel-by-channel  analysis  is  possible,  as  follows. 

The  packet  delay  is  defined  as  the  time  which  a 
packet  spends  in  the  network  from  its  entry  until  it 
reaches  its  destination.  The  average  packet  delay  is 
denoted  as  T.  Let  Zik  be  the  average  delay  for  those- 
packets  whose  origin  is  IMP  j  and  whose  destination  is 
IMP  k.  We  assume  a  Poisson  arrival  process  for  such 
packets  with  an  average  of  y,k  packets  per  second  and 
an  exponential  distribution  of  packet  lengths  with  an 
average  of  1/p  bits  per  packet.  With  these  definitions, 
if  y  is  the  sum  of  the  quantities  -y,*,  then21 

7’  =  E  —  3*  14; 

iM  y 

Let  us  now  reformulate  Equation  (4)  in  terms  of 
higl  efnimnl  «..-L*ys.  fm  U* 

quantities  for  the  ith  channel:  C\  as  its  capacity 
(bits/second) ;  X,  as  the  average  packet  traffic  it  carries 
(packets/second) ;  and  7\  as  the  average  time  a  packet 
spends  waiting  for  and  using  the  ith  channel.  By 
relating  the  (X*)  to  the  { y,t |  via  the  paths  selected  by 
the  routing  algorithm,  it  is  easy  to  sec  that21 

r-E-  t,  (5) 

i  y 

With  the  assumption  of  Poisson  traffic  and  exponential 
service  times,  the  quantities  7\  are  given  by  Equation 
(2).  For  an  average  packet  length  of  1/a  bits,  i~  1/pC,- 
seconds  and  thus 
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Thus  we  have  successfully  decomposed  the  analysis 
problem  into  a  set  of  simple  single-channel  problems. 


A  refinement  of  the  decomposition  permits  a  non¬ 
exponential  packet  length  distribution  and  use's  Equa¬ 
tion  (1)  rather  than  Equation  (2)  to  calculate-  7’,; 
as  an  approximation,  the-  Markovian  character  e>f  the 
traffic  is  assume-el  to  be  pre-se-rve-el.  Furthermore,  fe»r 
computes-  networks  we  include1  the  e-lTe-ct  of  propagation 
time-  anel  overhe'ad  traffic  to  obtain  the  following 
e-quation  for  average  packet  de-lay22'23 


T  =  K+ 


K/nC  i 


+  P.+K 


(7) 


He-re-,  1/p'  re-pre-se-nts  the  ave-rage-  length  of  a  Host 
packe-t,  anel  1/p  re-pre-se-nts  the-  average-  length  of  all 
packets  (inclueling  acknenvle-elgnients,  heaele-rs,  re-epmsts 
for  ne-xt  me-ssages,  parity  che-eks,  e-tc.)  within  the-  ne-t- 
wetrk.  The-  expression  l/MT'.  +  [(Xe//d \)  (mC,  — X,)]  +  /\ 
repre-se-nts  the  ave-rage  packe-t  ele-lav  on  the-  ith  channel. 
The  term  (X,/mC,)/(mC,-X,)  is  the-  ave-rage  time  :i 
packe-t  six-nds  waiting  at  the  IMP  for  the-  ith  channel  to 
be-come-  available-.  Since-  the  packe-t  must  compete-  with 
acknowledgments  anel  other  eiverhe-ael  traffic,  the- 
overall  ave-rage*  packe-t  le-ngth  1/p  appe-ars  in  the 
t-xpre-ssiem.  The  te-rm  l/n'C ,  is  the-  time-  re-epiireel  te> 
transmit  a  packet  of  average-  length  //•  Finally:  K  is 
the-  nodal  proce-ssing  time-,  assume-d  constant,  and  for 
the-  ARPA  IMP  approximately  e-qual  tei  0.3a  ms; 
Pi  is  the-  propagation  time  on  the  ith  channe-1  (about 
20  ms  for  a  3000  mile  channel) . 

Assuming  a  relatively  homogene-ous  se-t  eif  (',  and 
no  inelividual  term  in  the-  exnression  feir  delay  will 
dominate-  thee  summation  until  the-  flenv  X,  n  in  one- 
channel  (say  channel  ?,)  approache-s  the  capacity 
At  that  point,  the  te-rm  T,c,  and  he-nce-  T  will  grow 
f  .  ft  xptT  si.eTi f.iT  4s  Wit  u  (!ufuiiiale*Pbw 
one  (or  meire-)  te-rms  and  exhibits  a  thre-sholel  behavior. 
Prior  to  this  threshold,  T  re-mains  re-latively  constant. 

The-  accuracy  of  the-  time-  de-lay  mode-1,  as  we-ll  as  this  . 
thre-sholel  phe-nomenon  was  elemonstrate-d  on  a  19  noele- 
network14  and  on  the  ten  noele  ARPA  nert  derive-el  frejm 
Figur--  1(c)  by  deleting  the-  rightmost  five-  IMPs. 
Using  the-  routing  proeeelure  de-scribcd  in  the  last 
section2*  and  e-qual  traffic  between  all  noele  pairs,  the- 
channe-1  flows  X,  were  founel  for  the  ten  noele-  ne-t  and 
the  ele-lay  curve-s  shown  in  Figeire-  3  were  obtained. 
Curve-  A  was  obtained  with  fixe-el  1000  bit  packets,* 
while-  curve  B  was  generate-el  for  e-xpone-ntially  elis- 
tribute-d  variable-  h-ngth  jiacke-ts  with  ave-rage-  size-  eif 
500  bits.  In  both  cases  A  anel  B,  all  eive-rhe-ael  factors 
we-re-  ignore-el.  Ne>te-  that  the-  ele-lay  remains  small  until  a 


•  In  i'iisc  A,  the-  iippliration  uf  Kfiniitiim  (I)  allows  f- >r  nmsl:int 
packet  lengths  (i.e.,  zero  variance-). 
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total  throughput  slightly  greater  than  400  kilobits/ 
second  is  re* '  el.  The  delay  then  increases  rapidly. 
Curves  C  a. .a  lJ  respectively  represent  the  same 
situations  whe:  the  overhead  of  136  bits  per  packet  and 
per  RFNM  and  152  bits  per  acknowledgment  are 
included.  Notice  that  the  total  throughput  per  IMF’ 
is  reduced  to  250  kilobit  s/second  in  case  C  and  to 
approximately  200  kilobits/second  in  case  I). 

In  the  same  figure,  we  have  illustrated  with  x’s  the 
results  of  a  simulation  performed  with  a  realistic 
routing  and  metering  strategy.  The  simulation  omitted 
all  network  overhead  and  assumed  fixed  lengths  of 
1000  bits  for  all  packets. 

It  is  difficult  to  develop  a  practical  routing  and  flow 
control  procedure  that  will  allow  each  IMP  to  input 
identical  amounts  of  traffic.  To  compare  the  delay 


curve  .4  with  the  (mints  obtained  by  simulation,  the 
curve  should  actually  be  recomputed  for  the  slightly 
skewed  distribution  that  resulted.  It  is  notable  that  the 
delay  estimates  from  the  simulation  (which  used  a 
dynamic  routing  strategy)  and  the  computation  (which 
used  a  static  routing  strategy  and  the.  time  delay  for¬ 
mula)  .are  in  close  agreement.  In  particular,  they  both 
accurately  determined  the  vertical  rise  of  the  delay 
curve  in  the  range  just  above  400  kilobits/second,  the 
formula  by  predicting  infinite  delay  and  the  simulation 
by  rejecting  the  further  input  of  traffic. 

In  practice  and  from  the  analytic  and  simulation 
studios  of  the  ARPANLT,  the  average  queueing  delay 
is  observed  to  remain  small  (almost  that  of  an  unloaded 
net)  and  well  within  the  design  constraint  of  0.2  seconds 
until  the  traffic  within  the  network  approaches  the 
capacity  of  a  cutset.  The  delay  then  increases  rapidly. 
Thus,  as  long  os  traffic  is  low  enough  and  the  routing 
adaptive  enough  to  avoid  the  premature  saturation  of 
cutsets  by  guiding  traffic  along  paths  with  excess  capacity, 
queueing  delays  are  not  significant. 


Channel  capacity  optimization 

One  of  the  most  difficult  design  problems  is  the 
optimal  selection  of  capacities  from  a  finite  set  of 
options.  Although  there*  are*  many  heuristic  approaches 
to  this  problem,  analytic  results  an*  relatively  scarce. 
(For  the  specialized  case  of  centralized  networks,  an 
algorithm  yielding  optimal  results  is  available.")  While 
it  is  possible  to  find  an  economical  assignment  of 
discrete  capacities  for,  say,  a  200  IMP  network,  very 
little  is  known  about  the  relation  between  such  capacity 
assignments,  message  delay,  and  cost. 

To  obtain  theoretical  properties  of  optimal  capacity 
assignments,  one  may  ignore  the  constraint  that 
capacities  are  obtainable  only  in  discrete  sizes.  In 
Reference  21  such  a  problem  was  posed  where  the 
network  topology  and  average  traffic  flow  were  a-unn-d 
to  be  known  and  fixed  and  an  optimal  match  of  capaci¬ 
ties  to  traffic  flow  was  found.  Also,  the  traffic  was 
assumed  to  be  Markovian  (Poisson  arrival-  and 
exponential  packet  lengths)  and  the  independence 
assumption  and  decomposition  method  were  applied. 
For  each  channel,  the  capacity  C,  was  found  which 
minimized  the  average  message  delay  T,  at  a  fix.  d 
total  system  cost  I).  Since  A  ,/p  is  the  average  bit  rate  on 
the  ?th  channel,  the  solution  to  any  optimal  assignment 
problem  must  provide  more  than  this  minimal  capacity 
to  each  channel.  This  is  clear  since  both  Fquations  ((}) 
and  (7)  indicate  that  7\  will  become  arbitrarily  large 
with  less  than  (or  equal  to)  this  amount  of  capacity. 

It  is  not  critical  exactly  how  the  excess  capacity  is 
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assigned,  as  long  as  C,>\,  V-  Other  important  param¬ 
eters  and  insights  have  been  identified  in  studying  the 
continuous  capacity  optimization  problem.  For  ex¬ 
ample,  the  number  of  excess  dollars,  1)„  remaining 
after  the  minimum  capacity  X, / \i  is  assigned  to  each 
channel  is  of  great  importance.  As  D— *0,  the  average 
delay  must  grow  arbitrarily  large.  In  this  range,  the 
critical  parameters  become  p  and  n  where  p  =  y/pC  is 
the  ratio  of  the  rate  y/p  at  which  bits  enter  the*  network 
to  the  rate  C  at  which  the  net  can  handle  bits  and 
n  =  \/y,  where  X  =  £x,  is  the  total  rate  at  which  packets 
flow  within  the  net.  The  quantity  p  represents  a  dimen¬ 
sionless  form  of  network  “load”  whereas  n  is  easily 
shown  to  represent  the  average  path  length  for  a 
packet. 

As  the  load  p  approaches  1/n,  the  delay  T  grows  very 
quickly,  and  this  point  p=  1/n  represents  the  maximum 
load  which  the  network  can  support.  If  capacities  are 
assigned  optimally,  all  channels  saturate  simultaneously 
at  this  point.  In  this  formulation  n  is  a  design  parameter 
which  <  )ends  upon  the  topology  and  the  routing 
procedure,  while  p  is  a  parameter  which  depends  upon 
the  input  rate  and  the  total  capacity  of  the  network. 
In  studying  the  ARPANET’3  a  closer  representation 
of  the  actual  tariffs  for  high  speed  telephone  data 
channels  used  in  that  network  was  provided  by  setting 
D~  22,  d,C,  where  0<a< 1*  This  approach  requires 
the  solution  of  a  non-linear  equation  by  numerical 
techniques.  On  solving  the  equation,  it  can  be  shown 
that  the  packet  delay  T  varies  insignificantly  with  a 
for  .3<a<  1.  This  indicates  that  the  closed  form 
solution  discussed  earlier  with  a=l  is  a  reasonable 
approximation  to  the  more  difficult  non-linear  problem. 
These  continuous  capacity  studies  have  application  to 
general  network  studies  (e.g.,  satellite  communications)33 
and  are  under  continued  investigation.’5’6  46 

In  practice,  the  selection  of  channel  capacities  must 
be  made  from  a  small  finite  set.  Although  some  theo¬ 
retical  work  has  been  done  in  this  case  by  approxi¬ 
mating  the  discrete  cost -capacity  functions  by 
continuous  ones,  much  remains  to  be  done.13  ’5  Because 
of  the  discrete  capacities  and  the  time  varying  nature 
of  network  traffic,  it  is  not  generally  possible  to  match 
channel  capacities  to  the  anticipated  flows  within  the 
channels.  If  this  were  possible,  all  channels  would 
saturate  at  the  same  externally  applied  load.  Instead, 
capacities  are  assigned  on  the  basis  of  reasonable 
estimates  of  average  or  peak  traffic  flows.  It  is  the 
responsibility  of  the  routing  procedure  to  permit  the- 
traffic  to  adapt  to  the  available  capacity.14  Often  two 


•  Of  course  the  tariffs  reflect  the  discrete  nature  of  available 
channels.  The  use  of  the  exponent  a  provides  a  continuous  fit 
to  the  discrete  cost  function.  For  the  ARPANET,  a=. 8. 


IMP  sites  will  engage  in  heavy  communication  and 
thus  saturate  one  or  more  critical  network  cutsets.  In 
such  cases,  the  routing  will  not  he  able  to  send  addi¬ 
tional  flow  across  these  cuts.  The  network  will  theretore 
experience  “premature”  saturation  in  one  or  a  small  set 
of  channels  leading  to  the  threshold  behavior  described 
earlier. 

DISCUSSION 

A  major  conclusion  from  our  experience  in  network 
design  is  that  message  switched  networks  of  the  ARPA 
type  are  no  longer  difficult  to  specify.  They  may  be 
implemented  straightforwardly  from  the  specifications; 
they  can  be  less  expensive  than  other  currently  available 
technical  approaches;  they  perform  remarkably  well  as 
a  communication  system  for  interconnecting  time¬ 
sharing  and  batch  processing  computers  and  can  be 
adapted  to  directly  handle  teletypes,  displays  and  many 
other  kinds  of  terminal  devices  and  data  processing 
equipment.16  30 

The  principal  tools  available  for  the  design  of  net¬ 
works  are  analysis,  simulation,  heuristic  procedures, 
and  experimentation.  Analysis,  simulation  and  heuristics 
have  been  the  mainstays  of  the  work  on  modeling  and 
topological  optimization  while  simulation,  heuristic 
procedures  and  experimental  techniques  have  been  the 
major  tools  for  the  actual  network  implementation. 
Experience  has  shown  that  all  of  these  methods  are 
useful  while  none  are  all  powerful.  The  most  valuable 
approach  has  been  the  simultaneous  use  of  several  of 
these  tools. 

Each  approach  has  room  for  considerable  improve¬ 
ment.  The  analysis  efforts  have  not  yet  yielded  results 
in  many  important  areas  such  as  routing.  However,  for 
prediction  of  delay,  this  approach  leads  to  a  simple 
threshold  model  which  is  both  accurate  and  under¬ 
standable.  Heuristic  procedures  all  suffer  from  the 
problem  that  it  is  presently  unclear  how  to  select 
appropriate  heuristics.  It  has  been  the  innovative  use 
of  computers  and  analysis  that  has  made  the  approach 
work  well.  For  designing  networks  with  no  more  than  a 
few  hundred  I  Ml’s,  present  heuristics  appear  adequate 
but  a  good  deal  of  additional  work  is  required  for  net¬ 
works  of  greater  size.  Simulation  is  a  well  developed  tool 
that  is  both  expensive  to  apply  and  limited  in  the  overall 
understanding  that  it  yields.  For  these  reasons,  simula¬ 
tion  appears  to  he  most  useful  only  in  validating  models, 
and  in  assisting  in  detailed  design  decisions  such  as  the 
number  of  buffers  that  an  IMI’  should  contain  As  the 
size  of  networks  continues  to  grow,  it  appears  that 
simulation  will  become  virtually  useless  as  a  total  design 
tool.  The  ultimate  standard  by  which  all  models  and 
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conclusions  can  be  tested  is  experimentation.  Experi¬ 
mentation  with  the  actual  network  is  conceptually 
relatively  straight  forward  and  Very  Useful.  Although, 
experiments  an*  often  logistically  difficult  to  perform, 
the  •  -jii  j'ri luhle  mi  ilValU  tm  lehlilig  fnu>%, 

heuristics  and  design  parameters. 

The  outstanding  design  problems  currently  facing 
the  network  designer  are  to  specify  and  determine  the 
properties  of  the  routing,  flow  control  and  topological 
structure  for  large  networks.  This  specification  must 
make  full  use  of  a  wide  variety  of  circuit  options. 
Preliminary  studies  indicate  that  initially,  the  most 
fruitful  approaches  will  be  based  on  the  partitioning  of 
the  network  into  regions,  or  equivalently,  constructing 

a  Iittouik  Lj  CukulvCtifig  a  ixuiIiliiT  u[  fi  gilUtal 

networks.  To  send  a  message,  a  Host  would  specify 
hill  Lit  siilinuTi  n  «  1  I V  JraNhblioii  I  M  l' 

in  that  region.  No  detailed  implementation  of  a  large 
network  has  yet  been  specified  but  early  studies  of  their 
properties  indicate  that  factors  such  as  cost,  throughput, 
delay  and  reliability  are  similar  to  those  of  the  present 
ARPANET,  if  the  ARPA  technology  is  used.8 

Techniques  applicable  to  the  design  of  large  networks 
are  presently  under  intensive  study.  These  techniques 
appear  to  split  into  the  same  four  categories  as  small 
network  design  but  approaches  may  differ  significantly. 
For  example,  large  nets  are  likely  to  demand  the  place¬ 
ment  of  high  bandwidth  circuits  at  certain  key  locations 
in  the  topology  to  concentrate  flow.  These  circuits  will 
require  the  development  of  a  high  speed  IMP  to  connect 
them  into  the  net.  It  is  likely  that  this  high  speed  IMP 
will  have  the  structure  of  a  high  speed  multiplexor,  and 
may  require  several  cooperating  processors  to  obtain 
the  needed  computer  power  for  the  job.  Flow  control 
strategies  for  large  networks  seem  to  extrapolate  nicely 
from  small  network  strategies  if  each  region  in  the  large 
network  is  viewed  as  a  node  in  a  smaller  network. 
However,  this  area  will  require  additional  study  as  will 
the  problem  of  specifying  effective  adaptive  routing 
mechanisms.  Recent  efforts  indicate  that  efficient 
practical  schemes  for  small  networks  will  soon  be 
available.  These  schemes  seem  to  be  applicable  for 
adaptive  routing  and  flow  control  in  networks  con¬ 
structed  from  regional  subnetworks.  The  development 
of  practical  algorithms  to  handle  routing  and  flow- 
control  is  still  an  art  rather  than  a  science.  Simulation  is 
useful  for  studying  the  properties  of  a  given  heuristic, 
but  intuition  still  plays  a  dominant  role  in  the  system 
design. 

Several  open  questions  in  network  design  presently 
are:  (1)  what  structure  should  a  high  bandwidth  IMP 
have;  (2)  how  can  full  use  be  made  of  a  variety  of  high 
bandwidth  circuits;  (3)  how  should  large  networks  be 
partitioned  for  both  effective  design  and  operation; 


and  (4)  what  operational  procedures  should  large 
networks  follow?  Much  work  has  already  been  done  in 
tln^r  art  ate  but  mtiuh  mor,  r.  r.i.VltiS  T..  !,i  d.,i„  .  W 
expect  substantial  progress  to  be  achieved  in  (he  next 
Irw  |'i*Ahk  «»l  tmkiiUnirU^  Mr  i lift****}  ■ 
of  the  properties  of  message  switched  networks  of 
all  sizes. 
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ABSTRACT 

...  **eent  advances  In  computer  communication* 
•ro  diicuiied,  includmgi  1)  omputer  traffic  char- 
ta  ,be  c»,e  of  >«>»>*  holding  time  repre- 
I  *fc  41.  f  *'lentiflc  »PP‘icationi,  and  in  the  case  of 

Vml!  r*‘  rf,entln*  the  Inquiry-re. pona. 
Uf  mST  1  2  channel  error  characteristic* 

*p**d  VL  ce  b,nd  d,t»  tranamia.ion  on  the 
,  twitched  telecommunication  network,  and  of  the  low 
•peod  channel  at  a  rate  of  J00  bit./.ect  3)  optimal 
|  fixed  menage  block  rixe  for  communication  sys- 
item.  using  error  detection  and  retransmission a. 

I  41  ''I.*!'  f*ndom  or  bur«  «tror  channel)! 

DiJi.lal'kJ »1  multipUxing  (asynchronous  Time 
itiTi!!1 °“  Mu,tlp,exlng*i  5‘  loop  *y*tem«  and  6)  secur¬ 
ity  ‘B  computer  communication*.  New  area*  needimt 
further  investigation  are  included.  * 

1.  INTRODUCTION  | 

Lm.A.i:rput*!:-'on,-unie*t,on  *uch  a* 

•haring  and  diatributed  computer  system* 

i^j^tonurwie^arn<ltCOrIiPleX^ty'  ,h*  problem  of  being 
h-eol  Under,Und  *nd  to  predict  system  behavior 

tZst  iTee^'/f*  8LV  in,P°rt«n*.  It  become,  clear 
aynthe.iae  a  system  that  meet, 
efl  »**  1  d  p*rforn“nce  criteria  at  minimum 
It^JV.  computing,  communication  and  operation, 
nrf|k!r  pr°blem  *r*»*  need  to  be  studiedi  first, 
mS.t?.Vi  *C  n*  problem  between  computer  and  com- 
'econd,  the  relationship,  among 
communication  traffic  sources,  channel,  and  com- 
piter  resource  allocation  mechanisms. 

the..™*  <!!^^iCUltl*,  in,  ,tUdylng  *nd  understanding 
I  e  problems  aret  l)  computer  designers*  lack  of 

uTuoH' e'n.  HeCe0eT'TnlwC*Vtn  ‘«chn°“>*y>  2>  commun¬ 
ication  engineers  lack  of  knowledge  in  computer 

technology1  and  I)  the  lack  of  tool,  and  model,  with 

which  to  knalyte  the  behavior  of  these  complex 

•ystems.  The  fir.t  two  difficuitie.  may  be  resolved 

hy  exchange  of  information  between  computer 

designers  and  communication  specialists.  The  third 
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difficulty  may  be  remedied  by  periodically  summari¬ 
sing  important  research  related  to  computer- 
communication  systems  which  is  scattered  through¬ 
out  various  journals,  conference  proceedings  and 
technical  reports.  In  this  paper  we  aim  at  the  last 
objective. 

U.  RECENT  ADVANCES  IN  COMPUTER 
COMMUNICATIONS 

1.  Computer  Traffic  Characteristics 

It  ha.  become  apparent  that  real  progress  in 
modeling  and  analysis  depends  upon  more  than 
elegant  analytical  results  based  upon  convenient  but 
unsupported  assumptions.  Measurement  and  obser¬ 
vation  are  neededi  the  computer  traffic  character¬ 
istics  of  in-house  time-sharing  systems  has  been 
undertaken  by  the  Bell  Telephone  Laboratories  to 
obtain  estimates  of  system  variables.  Two  types 
of  system*  under  study  arei  long  holding  Lme 
(connect  to  disconnect)  and  short  holding  time.  Long 
holding  time  is  characteristic  of  business  and 
scientific  applications  which  require  extensive 
computation)  a  holding  time  typlcaiiy  of  i5  to  30 
minutes.  Short  holding  time  is  characteristic  of 
inquiry-response  systems  such  as  on-line  banking, 
credit  bureau  and  production  control  which  have 
holding  times  of  a  few  seconds  to  one  or  two  minutes. 

Jackson  and  Stubbs  ( 1 )  and  Fuchs  and  Jackson 
I  IfJ  have  reported  the  results  of  long  holding  time. 

They  show  that  the  volume  of  computer-to-user 
traffic  is  an  order  of  magnitude  higher  than  that  of 
user-to-computer  traffic.  The  interarrival  time 
between  messages  can  be  approximated  by  an 
exponential  distribution)  that  is,  the  stream  of 
messages  can  be  assumed  to  constitute  a  Poisson 
process.  Furthermore,  the  length  of  me ssagr • 
can  be  satisfactorily  approximated  bv  the  geometri¬ 
cal  distribution.  During  the  cal!  interval,  the  use- 
is  active  only  5!i  of  the  time  and  the  computer  is 
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active  i bout  10%  of  the  time.  Thue,  the  channei  ia 
Idle  for  a  significant  portion  of  the  holding  time. 

The  traffic  characteristics  of  short  holding  time 
are  reported  on  by  Dudick,  Fuchs  and  Jackson  [Jj. 
Ths  measured  results  from  four  such  systems 
reveal  that  user  send  time  (the  total  amount  of 
time  during  which  user  characters  are  being  trans¬ 
mitted)  Is  less  than  15%  of  the  holding  time.  This 
parameter  is  important  to  the  design  of  statistical 
multiplexors.  The  character  interarrival  time s 
can  be  represented  as  a  sum  of  two  gamma  distri¬ 
butions;  the  number  of  user  segments  per  call  end 
computer  segments  per  call,  can  be  represented 
by  a  geometrical  distribution.  These  measure¬ 
ments  and  estimated  system  variables  not  only 
provide  us  with  insight  into  the  behavior  of  the 
syetem  and  shed  light  on  areas  thet  need  improve- 
ments«  but  are  essential  in  the  modeling  and  analy¬ 
sis  of  computer  communication  systems. 

1,  Channel  Error  Characteristics  • 

The  communication  channel  provides  the  links 
between  processors  and  terminals  *.nd  plays  an 
important  role  in  computer  communication  systems. 
Thus,  a  characterisation  of  channel  performance 
if  important  for  understanding  the  cause  of  errors, 
for  suggesting  the  possible  improvements  in  ihe 
design  of  transmlarlon  equipment  and  the  design  of 
efficient  error  control  systems,  and  for  panning 
optimal  computer  communication  systems. 

To  characterise  telephone  channel  error  per¬ 
formance,  a  survey  measurement  program  of  the 
telephone  network  to  determine  the  error  perform¬ 
ance  and  the  data  speed  capabilities  is  necessary. 

|  A  series  of  such  studies  started  by  the  Bell  Systems 
|  in  1956  was  directed  towards  this  goal,  and  several 
,  papers  have  been  published  on  this  subject  (4-8]. 

J  Here  we  shall  emphasise  recent  survey  results 
(1949-1970)  on  high  speed  voice  band  data  trans¬ 
mission  performance  on  the  switched  telecommuni¬ 
cation  network  (7)  and  low  speed  data  transmission 
performance  on  the  switched  telecommunication 
network  [6].  In  these  two  papers,  the  distributions 
of  error  per  call  are  given  on  a  bit,  buret  and 
block  basis.  Information  is  also  presented  on  the 
distribution  of  intervals  between  errors,  the 
Structure  of  burst  errors  and  the  number  of  errors 
in  blocks  of  various  sites.  Such  statistics  provide 
information  on  channei  reliability  and  are  also 
useful  In  the  design  of  efficient  error  control 
procedures. 

Xn  the  high  speed  voiceband  data  transmission 
channel,  toll  traffic  was  used  as  a  basis  for  the 
sampling  plan  which  resulted  in  the  selection  of 
approximately  600  dialed-toll  connections  between 
geographically  dispersed  local  switching  offices. 

Data  rates  of  1200,  2000,  1600  and  4800  bits/sec 
are  measured  on  the  Beil  System  switch  telecom¬ 
munication  network.  The  measured  results  show 
a  substantial  improvement  of  performance  in 
comparison  with  the  results  of  previous  surveys; 
for  example,  the  measured  results  for  operation 
at  1200  and  2000  bits/sec  show  that  approximately 
62%  of  the  calls  have  an  average  error  rate  of  i 
error  in  105  bits  or  better  over  short,  medium  . 


and  long  haul  calls:  while  the  1959  Alexander,  Gryb 
and  Nast  -urvey  [4j  shows  only  6  If*  of  their  test 
calls  reached  this  performance  level  (iO"*)  for 
operation  at  the  same  data  rate.  A  general 
tendency  for  performance  to  degrade  with  trans¬ 
mission  distance  has  been  noted.  These  results 
also  Indicate  that  impulse  noise  accounted  for  a 
Urge  pci  cent.ge  of  the  observed  errors. 

Low  .peed  dete  transmission  correspond,  to 
teletypewriters,  computer  ports  and  other  terminel 
devices  that  communicate  by  means  of  data  organ¬ 
ised  in  characters  (comprised  of  several  bits) 
using  start-stop  transmission  et  a  rate  of  )00 
bits/sec.  Character  error  statistics,  rether  than 
bit  error  statistics,  are  the  parameters  of  interest 
in  this  type  of  transmission  because  the  messege 
consists  of  a  display  (in  teletypewriters)  or  use 
characters  in  most  applications  (in  computers). 
Measurements  were  made  on  5)4  connections  with 
over  21  million  characters  (I  character  •  10  bits) 
transmitted.  Over  90%  of  the  low  speed  test  calls 
contained  about  36,  000  to  54.  000  characters.  A 
character  error  rate  of  10*' or  less  is  indicated 
for  approximately  78%  of  all  calls,  while  95%  of 
all  c-illa  have  a  lost  cheracter  rate  of  10*'  or  less. 
Errors  occurring  in  the  messeges  are  in  bursts 
rather  than  at  random.  The  number  of  character 
errors  in  a  block  increases  with  the  block  length. 
Since  this  is  the  first  report  on  low  speed  data,  no 
comparleon  with  any  previous  survey  is  possible. 
Further  analysis  of  the  statistics  will  give  insight 
into  the  causes  of  errors  which  in  turn  may  suggest 
approaches  to  improve  error  performance. 

),  Optimal  Fixed  Message  Block  Site 

The  message  output,  from  a  computer  are 
usually  in  strings  of  character,  or  bursts.  The 
variation  of  message  length  can  best  be  described 
by  a  probability  distribution.  For  ease  in  data 
handling  and  memory  management,  the  random 
message  length  Is  usually  partitioned  into  several 
fixed  slae  blocks.  Due  to  the  random  length  of 
the  message,  the  last  partitioned  block  usually  is 
not  filled  by  the  message  but  is  filled  with  dummy 
information. 

For  reasone  of  economics  and  reliability, 
error  detection  and  retransmission  is  employed 
in  many  data  communication  systems  [5,  9).  The 
optimal  block  sise  is  an  important  parameter  in 
the  design  of  such  systems.  From  the  acknowl¬ 
edgement  point  of  view,  it  is  desirable  to  select 
the  largest  possible  block  else.  Since  each 
message  block  requires  at  least  one  acknowledge¬ 
ment  signal,  the  fewer  the  number  of  blocks  needed 
for  a  message,  the  the  channei  capacity 
required  for  acknowls  —ments.  On  the  other 
hand,  since  a  larger  message  block  has  a  higher 
channei  waetege  due  to  the  last  unfilled  partitioned 
block,  and  also  has  a  higher  probebility  of  error, 
it  is  desirable  to  select  the  smallest  possible  block 
siae.  Thus  there  is  a  trade-off  in  selecting  the 
optimal  block  sise. 
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Kucera(lO),  Balkovlc  and  Muench  ( i  i )  and 
Kirlin  [12]  have  studied  tht*  optimal  message  block 
sis*  for  the  error  detection  and  retransmission 
system  that  maximises  transmission  efficiency. 

Chu  [13]  considers  an  additional  important  param¬ 
eter  -  the  message  (file)  length  -  in  determining 
the  fixed  me  a  sage  block  site,  which  significantly 
effects  the  selection  of  the  optimal  block  sise. 

His  model  considers  average  message  (file)  length, 
message  length  distribution,  channel  error  char- 
i  acteristics  (random  error  and  burst  error),  over¬ 
head  for  addressing,  error  control  and  acknowl¬ 
edgement  delay.  His  criterion  for  optimality  is  to 
minimise  the  time  wasted  in  acknowledgements, 
retransmissions  and  the  waste  in  the  last  unfilled 
j-  block. 

i  M.llt  ••  M  III  ».  ■•••  I'  t’  I- 

4.  Statistical  Multiplexing 

Multiplexing  i(  commonly  uard  to  »h»re  and 
j  to  efficiently  utiliee  a  communication  channel. 
Currently  data  multiplexing  has  taken  two  format 
Frequency  Division  Multiplexing  (FDMl  and 
Synchronous  Time  Division  Multiplexing  (STDM) 
commonly  known  as  time  division  multiplexing. 
Frequency  Division  Multiplexing  divides  the 
channel  bandwidth  into  several  subchannels  such 
that  the  bandwidth  of  each  subchannel  la  greater 
than  that  required  for  a  message  channel.  Because 
of  the  need  to  employ  ‘guard  banda"  to  prevent 
data  aignaia  from  each  of  the  data  channel*  from 
feeding  into  adjacent  channel*  and  because  of  the 
relatively  poor  data  transmission  characteristics 
of  the  voice  band  hannet  near  the  edges  of  its 
bandwidth,  FDM  does  not  make  aa  efficient  uae  of 
the  voice  band  a  a  does  STDM. 

In  STDM,  each  user  (terminal)  ia  assigned  s 
fixed  time  duration  or  time  slot  on  the  communl- 
cation  channet  for  the  transmission  of  messages 
|  from  terminals  to  computer.  The  multiplexing 
apparatus  scans  the  set  of  users  in  a  round  robin 
faahlon.  After  one  ueer'  a  time  duration  has 
elapsed,  the  channel  la  twitched  to  another  user. 
With  appropriately  designed  synchronous  operation, 
required  buffering  can  be  limited  to  one  character 
par  terminal.  Addressing  la  usually  not  required 
•Ince  the  user  I*  identified  by  the  time  slot  posi¬ 
tion.  The  STDM  technique,  however,  also  has 
.  certain  disadvantages,  it  ia  inefficient  in  channel 
utilisation  to  permanently  assign  s  segment  of 
bandwidth  that  la  utilited  only  a  portion  of  the 
time.  Statistics  collected  fr"m  several  typical 
operating  time  sharing  eyelet...  [i]  showed  that 
during  a  call  (connect  to  diaconnectl,  the  user-to- 
computer  traffic  (or  the  long  holding  time  case  is 
active  only  5%  of  the  time.  Thus  STDM  would  be 
very  inefficient  in  channel  utilisation  in  such  an 
environment  alnce  STDM  allocates  a  time  clot  to 
each  ueer  independent  of  hie  activity.  In  order  to 
increase  channel  utilisation,  statistical  multi¬ 
plexing  or  Asynchronoue  Time  Division  Multi¬ 
plexing  (ATDM)  has  been  proposed  (14,  15)  for 
computer  communications.  The  basic  Idea  is  to 
twitch  from  one  ueer  to  another  user  whenever 
the  former  ia  idle  and  the  latter  ready  to  transmit 
data.  Thus  the  data  is  asynchronously  or  statisti¬ 
cally  multiplexed  with  respect  to  tl.e  users.  With 
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such  an  arrangement,  each  user  would  be  granted 
access  to  the  channel  only  when  he  hat  a  message 
to  transmit.  The  crucial  attributes  of  such  a  multi¬ 
plexing  technique  arei  I)  an  address  it  required 
for  each  transmitted  message  and  2)  buffering  ia 
required  to  hindle  atatiatlcal  fluctuations  in  the 
input  traffic. 

The  dal*  structure  for  messages  forming  the 
input  to  the  multiplexor  buffer  can  be  classified 
into  four  ca'egorlesi  constant  length  messages; 
random  length  messages;  mixed  (conxtant  and 
random  length)  meaaagex;  i*ndom  length  message 
segmented  into  fixed  sise  blocks.  The  constant 
length  message  input  correspond*  to  teletype  (TTY) 

I  input,  ea  h  user  types  in  one  character  at  a  time. 
Tho  random  length  message  input  corresponds  to 
paper  tap.'  input,  cathode -ray  tube  (CRT)  input  or 
computer  i.iput,  The  mixed  message  Input  corre¬ 
spond*  to  tisffic  from  a  mixture  of  type*  of  input 
terminals  such  aa  CRT,  TTY,  elc.  For  eaae  in 
data  handling  and  memory  management,  random 
length  message*  are  often  segmented  into  fixed 
else  blocks  which  correspond  10  the  last  type  of 
data  structure.  Since  messages  have  random 
length,  the  Usl  block  of  a  met  ge  usually  cannot 
be  entirely  filled.  Aa  a  result,  for  a  given  tralfic 
intensity,  this  type  of  data  structure  requires  a 
larger  buffer  than  that  of  random  length  messages 
which  are  not  segmented  ( 14).  The  buffer  behavior 
of  theae  (our  types  of  data  structures  have  been 
analysed  by  finite  watting  room  queueing  models 
[14-17]. 

The  buffer  behavior  of  a  statistical  multiplexor 
for  mixed  message  inputs  lies  between  that  of  con¬ 
stant  length  and  random  length  messages  (17).  The 
output  process  of  a  statistical  multiplexor  has  beer, 
studied  by  Pack  [!•]. 

The  demultiplexor  distributes  messages  to 
appropriate  destinations  according  to  their  message 
addresses.  Thus,  the  behavior  of  th*.  demulti¬ 
plexing  buffer  not  only  depends  on  traffic  intensity 
but  also  on  traffic  scheduling  to  venous  de stinatiens. 
In  the  case  of  a  time  sharing  system,  message 
scheduling  ts  determined  by  the  job  scheduling 
algorithm  of  the  computer  operating  system.  In 
the  case  of  distributed  computer  lyitrmi,  message 
scheduling  is  influenced  by  the  message  routing 
algorithm.  A  simulation  study  of  the  demulti¬ 
plexing  buffer  behavior  revealed  that,  for  a  given 
input  traffic  volume,  the  best  buffer  behavior  can 
be  achieved  by  scheduling  an  equal  amount  of  traffic 
to  each  deatination  (19).  Hence  there  it  a  close 
relationship  among  demulttpleKing  system  perform¬ 
ance  and  scheduling  algorithms  |20]  in  computer 
operating  systems  and/or  message  routing  algor¬ 
ithms.  Further  research  in  this  direction  would 
be  desirable.  Results  obtained  In  this  area  will 
be  essential  in  the  joint  optimisation  of  the  overall 
performance  of  such  computer  communication 
systems. 

Buffering  is  required  to  provide  error  control 
and  message  scheduling  which  are  two  important 
functions  in  computer  communication  systems. 

Since  statistical  multiplexing  requires  buffering  to 
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hudU  statistical  fluctuations,  the  multiplexing 
buffer  can  also  be  uaed  for  thee*  function*. 

From  three  studies  we  conclude  that  in  an 
ATOM  system,  an  acceptable  buffer  overflow 
probability  can  be  achieved  by  a  reaeonablc  buffer 
eiaej  the  expected  queueing  delay  Is  very  small  and 
acceptable  for  most  applications.  Hence,  ATOM 
or  statistical  multiplexing  Is  a  feasible  technique 
Hot  data  communications.  Furthermore  ATOM 
greatly  improves  the  transmission  efficiency  and 
system  organisation.  We  have  constructed  a 
statistical  multiplexor  at  UCLA,  Our  preliminary 
experience  has  shown  that  the  gain  tn  communi¬ 
cation  cost,  especially  in  long  distance  trans- 
►  mission,  by  employing  ATOM  in  computer  commun¬ 
ication  could  far  outweigh  costs  of  overhead  in 
addressing  and  storage  for  buffering.  Statistical 
multiplexing  should,  therefore,  have  high  potential 
(or  uae  In  future  computer-communication  systems. 

5.  Loop  Systems 

A  special  distributed  computer  system  archi¬ 
tecture  of  considerable  recent  Interest  la  the  loop 
(ring)  system.  Thl«  type  of  system  connects  all 
terminals  and/or  computers  by  a  common  bus  or 
loop.  The  major  advantages  are  the  simple  routing 
algorithm  and  ease  in  control  of  Information. 

Farmer  and  Newhall  (21)  proposed  and  constructed 
a  loop  system  with  bursty  traffic  which  connects 
various  devices  such  as  teletype,  plotter,  cathode- 
ray  tube  display,  disk  control  unit  and  computer 
togethar  Yuen,  et  al.  (22)  presents  some  approx  - 
imata  results  on  th<  traffic  behavior  of  this  dis¬ 
tributed  loop  eystem.  Pierce  (2)]  proposed  a 
hierarchy  of  Interconnected  loop  systems  with 
random  length  messages  segmented  into  fixed  siae 
blocks,  and  provides  a  scheme  for  transferring 
information  among  the  various  levels  of  loops. 
Koahelm  and  Melster  (24)  analysed  such  a  hier¬ 
archical  system.  Hayes  and  Sherman  (25)  studied 
ihe  message  delay  due  to  buffering  for  a  single 
I  loop  system.  The  data  source  Is  assumed  to  be  of 
a  bursty  mature.  The  traffic  generated  by  each 
user  Is  assumed  to  be  Identically  distributed  with 
uniformly  distributed  destinations  Konhelm  and 
Mslster  [26]  studied  the  loop  system  as  a  priority 
service  system.  Messages  may  enter  the  system 
at  any  Input  terminal  located  on  the  loop.  Priority 
Is  assigned  on  the  basis  of  position  on  the  loop; 
the  terminal  closest  to  the  computer  has  highest 
priority  and  priorities  decrease  with  '  distance1 
from  the  computer.  The  stationary  queue  lengths 
and  average  virtual  waiting  time  are  calculated. 
Spragins  [27)  has  calculated  the  waiting  time  of 
this  priority  loop  system  with  Poisson  arrival 
process.  Kaye  [26j  has  obtained  the  mean  and 
variance  of  menage  waiting  time  and  proportions 
of  blocked  messages  of  fixed  length  messages  of 
a  loop  system.  Farber  and  Larson  (29)  have  pro¬ 
posed  a  loop  system  using  fixed  message  length 
similar  to  that  of  Pierce  except  messages  are 
addresses  to  processes  rather  than  processors. 
Further,  messages  can  be  only  removed  at  their 
origin. 


Research  rosults  have  ahown  that  the 
acceptable  queueing  delay  can  be  achieved  when  the 
traffic  tn  a  loop  system  ts  under  careful  control. 

One  of  the  shortcomings  of  such  a  system  however. 
Is  system  reliability.  Further  study  is  needed  In 
determining  the  reliability  performance  of  loop 
systems  and  ways  of  improving  system  reliability. 
Such  studiea  will  enable  us  to  compare  the  cost- 
performance  of  loop  systems  with  that  of  other 
systems. 

6,  Security  tn  Computer  Communications 

With  the  growth  In  the  use  of  remote  terminal 
devices  and  time -sharing  systems  in  making 
Information  available  to  a  wide  variety  of  usera 
for  widely  diverse  applications,  the  problem  of 
computer  communication  security  becomes 
Increasingly  Important  (30-32).  The  communication 
channels  are  perhaps  the  most  vulnerable  compo¬ 
nents  of  the  computer  system  because  they  are 
easy  tn  access  by  methods  such  as  wire  tapping, 
picking  up  electromagnetic  pulses,  or  ’'piggy -back” 
entry  f 32 j.  The  nature  of  computer  communication 
security  is  quite  different  from  that  of  the  classical 
communication  security.  They  differ  at  least  in  the 
following  ways.  I)  Computer  files  usually  offer  a 
large  amount  of  data  to  work  on.  The  enemy  would 
have  to  know  exactly  what  type  of  Information  was 
In  each  file  (e.  g. ,  programs,  address  files, 
scientific  Information)  in  order  to  steal  It.  Z)  In 
computer  files,  all  records  are  usually  similar. 
Programs  have  a  high  rate  of  repeated  character¬ 
istics  (e.g.,  COBOL,  FORTRAN).  Also,  quite 
often  the  structure  of  the  computer  program  can  b* 
guessed  at.  All  these  similarities  can  help  an 
enemy  cryptanalyst  to  decod*  even  when  fairly 
sophisticated  cryptographic  techniques  are  used. 
Thus  new  methods  are  needed  to  provide  security 
in  computer  communications. 

The  use  of  logic  operation  (e.g.,  exclusive  OR) 
and  pseudo-random  numbers  presently  seems  to 
offer  the  greatest  poesibllity  for  computational 
cryptography.  A  unique  key  for  each  message  is 
generated  frem  the  pseudo-random  number  gen¬ 
erator,  and  the  message  then  performs  logical 
(exclusive  OR)  operations  wnth  the  key  to  produce 
the  Cipher  message.  The  reverse  operation 
deciphers  the  cipher  into  its  original  message. 

This  type  of  operation  is  usually  very  fast  and 
efficient  In  modern  computers  and  has  the  advantage 
of  good  security,  low  cost,  and  easy  changeability 
These  methods  have  been  explored  by  SVatrudjJJj. 
Krlshnamurthy,  [34]  and  Tassel.  (35) 

To  further  Increase  security  in  computer 
Information,  we  could  Introduce  multiple  pceudo- 
random  number  generators  to  achieve  higher 
levels  of  security.  For  example,  the  system  couti 
use  one  pseudo-random  number  generator  to  cipher 
the  message!  then  each  user  could  further  provide 
their  own  pseudo-random  number  generator  to 
provide  a  second  level  of  ciphering.  Another  way, 
perhaps  even  more  effretive.  It  to  use  an  expanded 
character  set  technique  to  break  down  the  statistical 
parameters  of  the  message,  such  as  frequency  of 
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•IngU  latter*,  diagram*,  vowel  percentage*.  etc. 
before  performing  the  logical  operation  with  the 
pseudo- random  number.  Our  preliminary  reaulta 
indicate  thi*  echeme  provide*  excellent  aecurlty 
performance.  Detailed  finding*  will  be  reported 
lathe  near  future. 

Sharing  of  file*  and  data  bate*  i*  not  only 
Important  in  application*,  but  alao  greatly 
iocrea***  flexibility  and  computer  capability. 
Security  and  protection  are  absolutely  necessary 
In  theae  facllitte*.  The  full  utilixation  of  theae 
*y*tem*  relle*  heavily  on  efficient  technique*  to 
provide  effective  information  aecunty.  Much 
work  atlll  need*  to  be  done. 

Other  advance*  which  have  not  been  included 
In  thl*  paper  but  have  Impact  on  computer  com- 
munlcatlon*  arei  error  control,  modulation, 
tran*mi**ion  medium,  (octal  and  regulatory 
policlea,  and  *tandardtiatton. 
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RAND  SATURATION  EXPERIMENT:  PRELIMINARY  RESULTS 


SUMMARY 

This  experiment  is  designed  to  study  delay  ond  throughput  characteristics 
of  a  part  of  the  ARPA  network  when  a  single  node  is  under  increasing  background 
loading.  In  particular,  RAND's  IMP  is  loaded  by  activating  IMP  message  generator 
fake  HOSTs  at  selected  IMPs  in  the  net  whose  messages  are  destined  for  the  RAND 
IMP  fake  HOST  discard. 

The  delay  and  throughput  experienced  by  a  single  site,  UCLA,  under  varying 
loading  conditions,  are  measured  and  the  results  discussed  in  the  light  of 
known  characteristics  of  the  IMP-IMP  message  switching  protocols  (sec  also 
references  B0N1822  and  HEART1970). 

EXPERIMENT  DESIGN 

At  the  time  this  experiment  was  performed,  RAND  was  the  only  site  in  the 
net  at  which  four  50KB  telephone  lines  were  connected  to  a  single  IMP  (sec 
map  on  next  page).  It  was  possible,  then,  to  transmit  as  much  as  200KB  into 
RAND's  IMP  over  these  four  lines. 

The  artificial  traffic  generators  in  each  IMP  are  capable  of  transmitting 
messages  to  a  single  site  over  a  single  logical  link.  The  messages  may  be 
transmitted  with  deterministic  inter-departure  times,  and  the  minimum  allowable 
delay  between  messages  is  the  time  for  the  message  to  be  sent  and  a  RFNM 
(Request  for  Next  Message)  to  be  returned  to  the  sending  HOST.  This  minimum 
delay  is  on  the  order  of  30  mseconds  for  adjacent  sites  and  increases  as 
the  number  of  IMPs  between  source  and  destination  IMPs  increases.  Message 
lengths  can  range  from  0  to  about  8000  bits  (exclusive  of  leader  and  padding). 
Details  of  the  IMP-IMP  protocols  and  message  generators  car,  be  found  in  (DBN1822). 

There  are  a  number  of  parameters  which  can  be  varied  in  an  experiment  of  thi6 
kind;  among  these  are  message  lengths,  number  of  message  generators  running, 
message  inter-departure  times,  topology  of  source  and  destination  IMP  data 
paths,  etc.  In  this  experiment,  the  nodes  of  the  network  are  divided  into 
four  classes:  measured  (UCLA  IMP),  receiving  node  (RAND  IMP),  senders,  unused. 
Messages  are  sent  from  UCLA  and  from  other  sending  nodes  to  RAND.  .Delays  and 
throughput  are  measured  for  UCLA  only. 
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Small  circled 
numbers  show  order 
in  which  traffic 
Generators  were 
turned  on  to  increase 
traffic  flow  to 
RAND.  Numbers 
inside  site  circles 
represent  ARPANET 
site  numbers. 
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The  length  of  messages  sent  from  UCLA  or  other  sending  sites  to 
RAND  varied  in  length  from  1  packet/  message  (960  bits)  to  4  or  8 
packets/message  (3840  and  7936  bits,  respectively).  Table  1  6hows 
the  various  combinations  of  message  lengths  used  in  the  9  different 
experiments. 


UCLA  OTHER  SENDERS 

1  p/m  1  p/m  p/m  -  packets/message 

1  4 

1  8 

4  1 

4  4 

4  8 

8  1 

8  4  < 

8  p/m  8  p/m 

Message  length  combinations  in  packets/message 
Table  1 

For  each  combination  of  message  lengths,  traffic  is  first  turned 
on  and  measured  from  UCLA  to  RAND  (see  ARPANET  map  on  page  2).  This 
is  accomplished  by  turning  on  both  the  artificial  traffic  generator 
at  UCLA's  IMP  and  a  cumulative  statistics  package  also  in  UCLA's  IMP 
which  accumulates  statistics  on  delays  and  throughputs  for  traffic 
leaving  UCLA's  IMP.  All  message  generators  are  run  with  RFNM  driven 
traffic  (i.e.  minimum  inter-departure  times).  Statistics  are  gathered 
for  six  minutes  (30  samples  of  12  second  cumulative  statistics),  and 
then  a  new  IMP  message  generator  is -started  for  the  next  six  minute 
period  (previous  generators  are  left  running).  In  the  map  on  page  2, 
the  order  in  which  IMP  message  generators  are  activated  for  increasing 
number  of  senders  is  shown  by  a  small  circled  number  next  to  a  site. 
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EXPERIMENT  REStn.TS 


n  the  figures  .hich  folio.,  the  roeuU.  of  the  experiment  arc  shown* 
on.  Important  point  to  keep  i„  mini  t.  that  th.  ..lay  „„d  ttawt 

c  eracter  .tie.  ob.erved  are  .pacifically  for  a  .curve  IHP  (ucu)  which  !. 

on  y  one  "hop"  away  from  the  de.tination  IMP  (RAND)  and  that  these  result. 

;;;t;rrcdf;° diffor  for  *  moa*urcd  •**  ***«»•  *°t  ^cent  to  thC 

..tination  (l.e.  more  than  one  "hop"  away).  Future  experiment.  .1,1 
investigate  th.  variation  in  throughput  and  delay  for  increasing  hop 
distance  between  sourc.  and  destination  node.. 

To  aid  th.  reader,  define  a  fe.  term,  .hich  are  used  in  the  figure. 

O’-  ““  «•  tb.  number  of  bit./aecond  actually  sent  out 

fr»  the  mea.urcd  .it.  (UCU).  Thi.  traffic  include.  retransmissions  and 
overhead  control  bit.,  of  which  may  not  have  been  accepted  by  th. 

de-tination  .u.  becau..  there  ...  not  enough  buffer  .pace  to  hold  th. 

t.  when  the  packet  arrived.  gFECTIVE  TH.ard.nim  computed  by 
Multiplying  th.  number  of  RFNM'.  received  by  th.  artificial  traffic 
..nerator  at  UCU  by  th.  length  of  the  me......  being  s,„t.  Effective 

throughput  doe.  not  include  overhead  bit.  and  retran.mi..ion.. 
igUND-TRIP  DELAY  i.  th.  delay  between  th.  depsrtur.  of  a  m....g.  from 

.  ertifical  traffic  generator  to  th.  time  a  RTN1I  i.  received  for  that 
"...age.  Por  multi-packet  ma...g„,  RFNM',  are  s.„t  only  after  all  packet, 
have  arrived  at  th.  destination  .It.  and  have  .tarted  into  the  receiving 

di  ,d.  ‘  ^  dl‘C‘rd  “  RA™)-  Ih'  gEU»/p^ET  l.  computed  by 

dividing  th.  Round-trip  delay  by  th.  me...,.  l,„8th  i„  ^Rcts. 

'lr,t,",rM  f“Ur"  b,1~  «"•“*«  t»«  re.ult.  for  the  case  that 
UCU  .end.  only  .Ingle  packet  me......  to  RAND  .hil.  other  .it.,  send 

1.  or  packet  mecages.  When  .a  ,Ue.  „ndlng  .In,u  ^ 

.....g..,  the  total  throughput  .horn  in  figure  1  tend,  to  drop  off  slightly 

foaoTZ?  *•  ‘ho*"  *•  the  effective  throughput 

«...«  ’  Ut  U  •“*“»  b,1~  -  ‘b‘  overhead  of  each 

*  ’  Th.  corresponding  delay,  .how,  ln  fi.ur,  3,  inere.Ie.  eH 
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For  the  case  that  other  sites  send  A  or  8  pocket  messages  while 
UCLA  sends  single  packet  messages,  there  is  much  more  noticeable 
interference.  In  the  case  of  8  packet  message  interference, 
comparison  of  the  total  and  effective  throughput  graphs  (figures 
1  and  2)  shows  that  after  5  sites  arc  sending,  UCLA  begins  to 
experience  retransmissions  as  well  as  simple  packet  overhead.  This 
shows  up  in  figure  3  as  a  non-linear  increase  in  dclay/packct. 

The  maximum  effective  data  rate  sustained  by  a  single  link 
is  shown  in  figure  2  as  roughly  27.5  KBS  (kilobits  per  second). 

Figures  4,  5,  6  consider  the  case  that  UCLA  is  sending  4  packet 
messages.  We  notice  in  figure  5  that  the  maximum  single  link 
data  rate  is  now  about  37  KBS.  The  increase  is  not  more  since 
the  RFNM  delay  has  not  decreased  to  one  fourth  of  its  single  packet 
message  value,  but  is  only  three  fourths  (26. /34.).  However,  there 
is  a  drastic  interference  between  multi-packet  messages,  as  we  see 
in  figure  4.  After  more  than  3  sites  are  sending  multi-packet 
messages,  the  total  throughput  drops  rapidly.  The  effective  throughput 
drops  ev.sn  faster  (see  figure  5).  The  peculiar  shape  of  the  curve 
in  figure  4  for  UCLA  sending  4  packet  messages  and  others  sending  8 
(i.e.  total  throughput  curve  increases  again),  can  be  explained 
by  the  following  argument.  Total  throughput  begins  to  drop  as 
Interference  Increases  because  there  is  contention  for  buffer  space 
and  the  RFM  delay  increaees.  After  a  while,  however,  the  sending 
site  begins  to  time-out  while  waiting  for  an  ACK  from  transmitted 
packets,  and  starts  to  re-transmit.  Retransmissions,  of  course, 
are  not  subject  to  the  "blocked  link"  convention,  and  thus  can 
create  traffic  in  excess  of  the  maximum  achievable  effective  traffic. 
(Figure  1  also  shows  this  effect). 

The  sudden  drop  in  throughput  (total  and  effective)  after  more  than 
three  multi-packet  senders  are  running  is  the  result  of  contention  for 
re-assembly  buffer  space  in  RAND's  IMP.  There  is  enough  re-assembly 
buffer  space,  for  three  full  multi-packet  messages.  The  buffers  are 
allocated  on  the  assumption  that  any  incoming  multi-packet  belongs 
to  a  message  containing  8  packets,  since  it  is  not  known  at  allocation 
time  how  long  the  multi-packet  really  is.  When  more  than  3  sites  send 
multi-packet  messages  to  RAND's  IMP,  there  is  strong  contention  for 
the  available  buffer  space. 
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Similar  results  are  shown  in  figures  8,  9,  and  10  for  the 
case  that  UCLA  sends  full  8  packet  multi-packet  messages.  The  maximum 
sustainable  effective  throughput  fo:  a  single  link  is  40.5  KBS. 

We  note  that  when  there  is  little  or  no  interfering  traffic,  the 
RFNM  delay/packet  drops  from  34  msec  to  25  msec  as  message  length 
increases  from  1  to  8  packets. 
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NOTE:  UCLA  SENDING  1  PACKET  MESSAGES 

X  -  OTHER  SITES  SENDING  1  PACKET  MESSAGES 
o  -  OTHER  SITES  SENDING  6  PACKET  MESSAGES 
♦  -  OTHER  SITES  SENDING  8  PACKET  MESSAGES 
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NOTE:  UCLA  SENDING  4  PACKET  MESSAGES 

X  -  OTHER  SITES  SENDING  1  PACKET  MESSAGES 
o  -  OTHER  SITES  SENDING  4  PACKET  MESSAGES 
♦  -  OTHER  SITES  SENDING  8  PACKET  MESSAGES 
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DELAY  (ROUND-TRIP)  PER  PACKET  IN  MILLISECONDS 


DELAY  AT  UCLA  VERSUS  NUMBER  OF  SEEDING  SITES 
FIGURE  6  i 


NOTE:  UCLA  SENDING  4  PACKET  MESSAGES 

X  -  OTHER  SITES  SENDING  1  PACKET  MESSAGES 
O  -  OTHER  SITES  SENDING  4  PACKET  MESSAGES 
*  -  OTHER  SITES  SENDING  8  PACKET  MESSAGES 
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NOTE:  UCLA  SENDING  8  PACKET  MESSAGES 
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INTRODUCTION 

Much  has  been  said  about  the  mechanics  of  the  ARPA 
Computer  Network  (ARPANET)  and  especially  about 
the  organization  of  its  communications  subnet. 1  •,1 4  •* 
Until  recently  the  main  effort  has  gone  into  the  imple¬ 
mentation  of  an  ARPANET  user-level  communications 
interface.  Operating  just  above  the  communications 
subnet  in  ARPANET  HOST  Computers,  this  ARPA¬ 
NET  interface  is  intended  to  serve  as  a  foundation  for 
the  organization  of  funetion-oriented  communiea- 
tions .*•’  Sec  Figures  1  and  2  for  our  view  of  a  computer 
system  and  the  scheme  for  user-level  process-to-proccss 
communications.  It  is  now  appropriate  to  review  the 
development  of  protocols  whieh  have  been  eonstrueted 
to  promote  particular  substantive  uses  of  the 
ARPANET,  namely  funetion-oriented  protocols. 

We  should  begin  this  brief  examination  by  stating 
what  we  mean  by  the  word  "protocol”  and  how  proto¬ 
cols  fit  in  the  plan  for  useful  work  on  the  ARPANET. 
When  we  have  two  processes  facing  eaeh  other  across 
some  communication  link,  the  protocol  is  the  set  of 
their  agreements  on  the  format  and  relative  timing  of 
messages  to  be  exchanged.  When  we  speak  of  a  proto¬ 
col,  there  is  usually  an  important  goal  to  be  fulfilled. 
Although  any  set  of  agreements  between  cooperating 
(i.e.,  communicating)  processes  is  a  protocol,  the  proto¬ 
cols  of  interest  arc  those  whieh  are  constructed  for 
general  application  by  a  large  population  of  processes 
in  solving  a  large  class  of  problems. 

In  the  understanding  and  generation  of  protocols 
there  are  two  kinds  of  distinctions  made.  Protocols  in 
the  ARPANET  are  layered  and  we  speak  of  high  or 
low  level  protocols.  High  level  protocols  are  those  most 
elosely  matched  to  functions  and  low  level  protocols 
deal  with  communications  mechanics.  The  lowest  level 
software  protocols  in  the  ARPANET  involve  reliable 


message  exchange  between  ARPANET  Interface 
Message  Processors  (lMPs).‘  ‘  A  high  level  protocol  is 
one  with  primitives  elosely  related  to  a  substantive  use. 
At  the  lowest  levels  the  contents  of  messages  are  un¬ 
specified.  At  higher  levels,  more  and  more  is  stated 
about  the  meaning  of  message  contents  and  timing.  The 
layers  of  protocol  are  shown  in  Figure  3. 

A  seeond  way  of  structuring  sets  of  protocols  and 
their  design  is  bound  up  in  the  word  factoring.  At  any 
level  of  protocol  are  sets  of  format  and  timing  rule.-* 
associated  with  particular  groupings  of  agreements.  In 
the  IMPS  we  find  eertain  protocols  pertaining  to  error 
handling,  while  others  to  flow  eontrol,  and  still  others 
to  message  routing.  At  the  ARPANET  s  user-le\  el 
communications  interface  there  are,  among  others, 
separable  protocols  associated  with  establishing  con¬ 
nections  and  logical  data  blocking.  These  protocols  do 
not  nest,  but  join  as  modules  at  the  same  level. 

Before  moving  on  to  consider  the  higher  level  fune¬ 
tion-oriented  protocols,  let  us  first  make  a  few  state¬ 
ments  about  underlying  protocols.  There  are  three 
lower  level  software  protocols  whieh  nest  in  support  of 
the  user-level  communications  interface  for  the  ARPA¬ 
NET-.  The  lowest  of  these  is  the  IMP-IMP  protocol 
which  provides  for  reliable  communication  among 
IMPs.  This  protocol  handles  transmission-error  detec¬ 
tion  and  correction,  flow  eontrol  to  avoid  message 
congestion,  and  routing.  At  the  next  higher  level  is  the 
IMP-HOST  protocol  whieh  provides  for  the  passage 
of  messages  between  HOSTs  and  IMPs  in  such  a  way 
as  to  ereate  virtual  communication  paths  between 
HOSTs.  With  IMP-HOST  protocol,  a  HOST  has 
operating  rules  whieh  permit  it  to  send  message^  to 
specified  HOSTs  on  the  ARPANET  and  to  be  informed 
of  the  dispensation  of  those  messages.  In  particular,  the 
IMP-HOST  protocol  constrains  HOSTs  in  their  trans¬ 
missions  so  that  they  ean  make  good  use  of  available 
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Figure  1 — Our  view  of  a  computer  system 


communications  capacity  without  denying  such  avail¬ 
ability  to  other  HOSTs. 

The  HOST-HOST  protocol,  finally,  is  the  set  of 
rules  whereby  HOSTs  construct  and  maintain  com¬ 
munication  between  processes  (user  jobs)  running  on 
remote  computers.  One  process  requiring  communica¬ 
tions  with  another  on  some  remote  computer  system 
makes  requests  on  its  local  supervisor  to  act  on  its 
behalf  in  establishing  and  maintaining  those  communi¬ 
cations  under  HOST-HOST  protocol. 

In  constructing  these  low  levels  of  protocol  it  vas  the 
intention  to  provide  user  processes  with  a  general  set 
of  useful  communication  primitives  to  isolate  them 
from  many  of  the  details  of  operating  systems  and 
communications.  At  this  user-level  interface  function- 
oriented  protocols  join  as  an  open-ended  collection  of 
modules  to  make  use  of  ARPANET  capabilities. 

The  communications  environment  facing  the  de¬ 
signers  of  function-oriented  protocols  in  the  ARPANET 


is  essentially  that  of  a  system  of  one-way  byte-oriented 
connections.  Technically  speaking,  a  "connection"  is  a 
pair:  a  “send  socket”  at  one  end  and  a  "receive  socket" 
at  the  other.  Primitives  provided  at  the  user-level 
interface  include: 

1.  Initiate  connection  flocal  socket,  foreign  socket), 

2.  Wait  for  connection  (local  socket), 

3.  Send,  Receive  (local  socket,  data), 

4.  Close  (local  socket), 

5.  Send  interrupt  signal  (local  socket). 

Processes  in  this  virtual  process  network  can  create 
connections  and  transmit  bytes.  Connections  arc  sub¬ 
ject  to  HOST-HOST  flow  control  and  the  vagaries  of 
timing  in  a  widely  distributed  computing  environment, 
but  care  has  been  taken  to  give  user  processes  control 
over  their  communications  so  as  to  make  full  u-e  of 
network  parallelism  and  redundancy.  The  kind  of 
agreements  which  must  be  made  in  the  creation  of 
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Figure  2— Two  communicating  prop  wo 


function-oriented  protocols  relate  to  rules  for  estab¬ 
lishing  connections,  to  the  timing  rules  which  govern 
transmission  sequences,  and  to  the  content  of  the  byte- 
streams  themselves. 

USE  OF  REMOTE  INTERACTIVE  SYSTEMS 

The  application  which  currently  dominates  ARPA¬ 
NET  activity  is  the  remote  use  of  interactive  systems. 
A  Telecommunications  Network  (TELNET)  protocol 
is  followed  by  processes  cooperating  to  support  this 
application.1  A  user  at  a  terminal,  connected  to  his 
local  HOST,  controls  a  process  in  a  remote  HOST  as  if 
he  were  a  local  user  of  the  remote  HOST.  His  local 
HOST  copies  characters  between  his  terminal  and 
TELNET  connections  over  the  ARPANET.  We  refer 
to  the  HOST  where  the  user  sits  as  the  uting  HOST, 
and  to  the  remote  HOST  as  the  tenting  HOST.  Sec 
Figure  4. 


At  the  using  HOST,  the  user  must  be  able  to  per¬ 
form  the  following  functions  through  his  TELNET 
user  process  (“uscr-TELNET"): 

1.  Initiate  a  pair  of  connections  to  a  serving  HOST, 

2.  Send  characters  to  the  serving  HOST, 

3.  Receive  characters  from  the  serving  HOST, 

4.  Send  a  HOST-HOST  interrupt  signal, 

5.  Terminate  connections. 

The  uscr-TELNET  needs  to  be  able  to  distinguish  be¬ 
tween  (I)  commands  to  be  acted  on  locally  and  (2) 
input  intended  for  the  serving  HO£T.  An  escajio  char¬ 
acter  is  reserved  to  mark  local  commands.  Conventions 
for  the  ARPANET  Terminal  IMP  (TIP)  urer- 
TELNET  are  typical.* 

In  most  using  IIOSTs,  the  above  functions  arc  pro¬ 
vided  by  a  uscr-TELNET  which  is  a  uscr-lrtd  program 
A  minimal  uscr-TELNET  need  only  implement  the 
above  functions,  but  several  additional  support  func- 
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Figure  3— The  layer*  of  protocol 


tions  are  often  provided  (c.g ,  saving  a  transcript  of  r 
session  in  a  local  file,  sending  a  file  in  place  of  ueur- 
typed  input,  repotting  whctlicr  various  llOSTs  nre  or 
have  been  up). 

In  the  serving  IIJST  it  is  desirable  that  a  process 
controlled  over  the  AitPAHKT  behave  as  it  would  if 
controlled  locally.  The  cleanest  way  to  achieve  this 
goal  is  to  generalize  the  terminal  control  portion  (TCP) 
of  the  operating  system  to  accept  ARPANET  terminal 
interaction.  It  is  unpleasant  to  modify  any  portion  of 
a  working  computer  system  and  modification  could  be 
avoided  if  it  were  passible  to  use  a  non-supervisor 
process  (c.g.,  “servcr-TELNET"  or  “LOGGER")  to 
perform  the  job  creation,  login,  terminal  input-output, 
interrupt,  and  logout  functions  in  exactly  the  same  way 


SERVING  HOST 


Figure  4 — Data  flow  for  remote  interactive  use 
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Figure  5 — Data  flow  scheme  for  server 


•s  a  direct  console  user.  Prior  to  the  development  of  the 
ARPANET,  no  operating  system  provided  these  func¬ 
tions  to  non-supervisor  processes  in  anywhere  near  the 
required  completeness.  Some  systems  have  since  been 
modified  to  support  this  generalised  job  control  scheme. 
See  Figures  5  and  6. 

Efforts  to  standardise  communications  in  the  TEL- 


NET  protocol  focused  on  four  issues:  character  set, 
echoing,  establishing  connections,  and  attention 
handling. 

The  chosen  character  set  is  7-bit  ASCII  in  8-bit 
fields  with  the  high-order  bit  off.  Codes  with  the  high- 
order  bit  on  arc  reserved  for  TELNET  control  func¬ 
tions.  Two  such  TELNET  control  function  codes  are 
the  “long-space”  which  stands  for  the  200  millisecond 
space  generated  by  the  teletype  BREAK  button,  and 
the  synchronization  character  (SYNCH)  discussed  be¬ 
low  in  conjunction  with  the  purpose  of  the  TELNET 
interrupt  rignal. 

Much  controversy  existed  regarding  echoing.  The 
basic  problem  is  that  some  systems  expect  to  echo, 
while  some  terminals  always  echo  locally.  A  set  of  con¬ 
ventions  and  signals  was  developed  to  control  which 
side  of  a  TELNET  connection  should  echo.  In  practice, 
those  systems  which  echo  have  been  modified  to  include 
provision  for  locally  echoing  terminals.  This  is  a  non¬ 
trivial  change  affecting  many  parts  of  a  serving  HOST. 
For  example,  normally  echoing  server  HOSTs  do  not 
echo  passwords  so  as  to  help  maintain  their  security. 
Terminals  which  echo  locally  defeat  tliis  strategy,  how- 


Figiire  6— Alternate  data  flow  scheme  for  a  server 
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Figure  7— Data  flow  end  processing  of  the  character 
input  atream 


ever,  and  some  other  protection  scheme  is  necessary. 
Covering  the  password  with  noise  characters  is  the 
usual  solution. 

The  HOST-HOST  protocol  provides  a  large  number 
of  sockets  for  each  HOST,  but  carefully  refrains  from 
specifying  which  ones  are  to  be  used  for  what.  To  estab¬ 
lish  communication  between  a  user-TELNET  and  a 
server-TELNET  some  convention  is  required.  The 
Initial  Connection  Protocol  (ICP)"1  is  used: 

!•  Connection  is  initiated  from  a  user-TELNET’s 
receive  socket  to  a  serving  HOST’s  socket  1 
(a  send  socket). 

2.  When  the  initial  connection  is  established,  the 
serving  HOST  sends  a  generated  socket  number 
and  closes  the  connection.  This  socket  number 
identifies  an  adjacent  socket  pair  at  the  serving 
HOST  through  which  the  user-TELNET  can 
communicate  with  a  server-TELNET. 

3.  TELNET  connections  are  then  initiated  be¬ 
tween  the  now  specified  pairs  of  sockets.  Two 
connections  are  used  to  provide  bi-directional 
communication. 

Note  that  socket  1  at  the  serving  HOST  is  in  use  only 
long  enough  to  send  another  socket  number  with  which 
to  make  the  actual  service  connections. 

One  of  the  functions  performed  by  a  terminal  control 
program  within  an  operating  system  is  the  scanning  of 
an  input  stream  for  attention  characters  intended  to 
stop  an  errant  process  and  to  return  control  to  the 
executive.  Terminal  control  programs  which  buffer  in¬ 
put  sometimes  run  out  of  space.  When  this  happens  to 
a  local  terminal's  input  stream,  a  “bell”  or  a  question 


mark  is  echoed  and  the  overflow  character  discarded, 
after  checking  to  sec  if  it  is  the  attention  character.  See 
Figure  7.  Phis  strategy  works  well  in  practice,  but  it 
depends  rather  strongly  on  the  intelligence  of  the  human 
user,  the  invariant  time  delay  in  the  input  transmission 
system,  and  a  lack  ef  buffering  between  type-in  and  at¬ 
tention  checking.  None  of  these  conditions  exists  for 
interactive  traffic  over  the  net:  The  serving  HOST  can¬ 
not  control  the  speed  (except  to  slow  it  down)  or  the 
buffering  within  the  using  HOST,  nor  can  it  even  know 
whether  a  human  user  is  supplying  the  input.  It  is  thus 
necessary  that  the  terminal  control  program  or  server- 
TELNET  not,  in  general,  discard  characters  from  a  net¬ 
work  input  stream;  instead  it  must  suspend  its  accept¬ 
ance  of  characters  via  the  HOST-HOST  flow  control 
mechanism.  Since  a  HOST  may  only  send  messages 
when  there  is  room  at  the  destination ,  the  responsibility 
for  dealing  with  too  much  input  is  thus  transferred  back 
to  the  using  HOST.  This  scheme  assures  that  no  charac¬ 
ters  accepted  by  the  using  HOST  are  inadvertently  lost. 
However,  if  the  process  in  the  serving  HOST  stops  ac¬ 
cepting  input,  the  pipeline  of  buffers  between  the  user- 
TELNET  and  remote  process  will  fill  up  so  that  atten¬ 
tion  characters  cannot  get  through  to  the  serving 
executive.  In  the  TELNET  protocol,  the  solution  to 
this  problem  calls  for  the  user-TELNET  to  send,  on 
request,  a  HOST-HOST  interrupt  signal  forcing  the 
server-TELNET  to  switch  input  modes  to  process  net¬ 
work  input  for  attention  characters.  The  server- 
TELNET  is  required  to  scan  for  attention  characters 
in  its  network  input,  even  if  some  input  must  be  dis¬ 
carded  while  doing  so.  The  effect  of  the  interrupt  signal 
to  a  server-TELNET  from  its  user  is  to  cause  the  buf¬ 
fers  between  them  to  be  emptied  for  the  priority  pro¬ 
cessing  of  attention  characters. 

To  flip  an  attention  scanning  server-TELNET  bark 
into  its  normal  mode,  a  special  TELNET  synchroniza¬ 
tion  character  (SYNCH)  is  defined.  When  the  server- 
TELNET  encounters  this  character,  it  returns  to  the 
strategy  of  accepting  terminal  input  only  as  buffer 
space  permits.  There  is  a  possible  race  condition  if  the 
SYNCH  character  arrives  before  the  HOST-HOST 
interrupt  signal,  but  the  race  is  handled  by  keeping 
a  count  of  SYNCHs  without  matching  signals.  Note 
that  attention  char,'  ;ters  are  HOST  specific  and  mav 
be  any  of  129  characters— 12S  ASCII  plus  “long 
space”  while  SYNCH  is  a  TELNET  control  character 
recognized  by  all  server-TELNETs.  It  would  not  do 
to  use  the  HOST-HOST  signal  alone  in  place  of  the 
signal-SYNCH  combination  in  attention  processing, 
because  the  position  of  the  SYNCH  character  in  the 
TELNET  input  stream  is  required  to  determine  where 
attention  processing  ends  and  where  normal  moth'  input 
processing  begins. 


lio 


Funetion-Oricnted  Protocols  for  ARPA  Computer  Network  277 


FILE  TRANSFER 

When  viewing  the  ARPANET  as  a  distributed 
computer  operating  system,  one  initial  question  is  that 
of  how  to  construct  a  distributed  file  system.  Although 
it  is  constructive  to  entertain  speculation  on  how  the 
ultimate,  automated  distributed  file  system  might  look, 
one  important  first  step  is  to  provide  for  the  simplest 
kinds  of  explicit  file  transfer  to  support  early  sub¬ 
stantive  use. 

During  and  immediately  after  the  construction  of  the 
ARPANET  user-level  process  interface,  several  ad  hoc 
file  transfer  mechanisms  developed  to  provide  support 
for  initial  use.  These  mechanisms  took  two  forms:  (1) 
use  of  the  TELNET  data  paths  for  text  file  transfer  and 
(2)  use  of  raw  byte-stream  communication  between 
compatible  systems. 

By  adding  two  simple  features  to  the  user-TELNET, 
text  file  transfer  became  an  immediate  reality.  By 
adding  a  “SCRIPT”  feature  to  user-TELNETS 
whereby  all  text  typed  on  the  user’s  console  can  be 
directed  to  a  file  on  the  user’s  local  file  system,  a  user 
need  only  request  of  a  remote  HOST  that  a  particular 
text  file  be  typed  on  his  console  to  get  that  file  trans¬ 
ferred  to  his  local  file  system.  By  adding  a  "SEND- 
FILE”  feature  to  a  user-TELNET  whereby  the  con¬ 
tents  of  a  text  file  can  be  substituted  for  console  type-in, 
a  user  need  only  start  a  remote  system’s  editor  as  if  to 
enter  new  text  and  then  send  his  local  file  as  type-in 
to  get  it  transferred  to  the  remote  file  system.  Though 
crude,  both  of  these  mechanisms  have  been  used  with 
much  success  in  getting  real  work  done. 

Between  two  identical  systems  it  has  been  a  simple 
matter  to  produce  programs  at  two  ends  of  a  connection 
to  copy  raw  bits  from  one  file  system  to  another.  This 
mechanism  has  also  served  well  in  the  absence  of  a  more 
general  and  powerful  file  transfer  system. 

Ways  in  which  these  early  ad  hoc  file  transfer  mech¬ 
anisms  are  deficient  are  that  (1)  they  require  explicit 
and  often  elaborate  user  intervention  and  (2)  they  de¬ 
pend  a  great  deal  on  the  compatibility  of  the  file  sys¬ 
tems  involved.  There  is  an  on-going  effort  to  construct 
a  File  Transfer  Protocol  (FTP)11-'*  worthy  of  wide 
implementation  which  will  make  it  possible  to  exchange 
structured  sequential  files  among  widely  differing  file 
systems  with  a  minimum  (if  any)  explicit  user  inter¬ 
vention.  In  short,  the  file  transfer  protocol  being  de¬ 
veloped  provides  for  the  connection  of  a  file  transfer 
user  process  (“user-FTP”)  and  file  transfer  server 
process  (“server-FTP”)  according  to  the  Initial  Con¬ 
nection  Protocol  discussed  above.  Sec  Figure  8.  A  user 
will  be  able  to  request  that  specific  file  manipulation 
operations  be  performed  on  his  behalf.  The  File  Trans¬ 
fer  Protocol  will  support  file  operations  including  (1) 


Figure  8 — Data  flow  for  fiJe  transfer 


list  remote  directory,  (2)  send  local  file,  (3)  retrieve  re¬ 
mote  file,  (4)  rename  remote  file,  and  (5)  delete  remote 
file. 

It  is  the  intention  of  the  protocol  designers  to  regu¬ 
larize  the  protocol  so  that  file  transfer  commands  can 
be  exchanged  by  consoles  file  transfer  jobs  engaged  in 
such  exotic  activities  as  automatic  back-up  and  dy¬ 
namic  file  migration.  The  transfers  envisioned  will  be 
accompanied  with  a  Data  Transfer  Protocol  (DTP)11 
rich  enough  to  preserve  sequential  file  structure  and  in 
a  general  enough  way  to  permit  data  to  flow  between 
different  file  systems. 

USING  THE  ARPANET  FOR  REMOTE 

JOB  ENTRY 

A  very  important  use  of  the  ARPANET  is  to  give  a 
wide  community  of  users  access  to  specialized  facilities. 
One  type  of  facility  of  interest  is  that  of  a  very  powerful 
number-cruncher.  Users  in  the  distributed  ARPANET 
community  need  to  have  access  to  powerful  machines 
for  compute-intensive  applications  and  the  mode  of 
operation  most  suited  to  these  uses  has  been  batch 
Remote  Job  Entry  (RJE).  Typically,  a  user  will  generate 
a  "deck”  for  submission  to  a  batch  system.  See  Figure  0. 
He  expects  to  wait  for  a  period  on  the  order  of  tens  of 
minutes  or  hours  for  that  “deck”  to  be  processed,  and 
then  to  receive  the  us-,  ally  voluminous  output  thereby 
generated.  See  Figure  10. 

As  in  the  ease  ot  file  transfer,  there  are  a  few  useful 
ad  hoc  ARPANET  RJE  protocols.  A  standard  RJE 
protocol  is  being  developed  to  provide  for  job  sub¬ 
mission  to  a  number  of  facilities  in  the  ARPANET. 
This  protocol  is  being  constructed  using  the  TELNET 
and  File  Transfer  protocols.  A  scenario  which  sketch'  s 
how  the  protocol  provides  the  RJE  in  the  simplest, 
most  explicit  way  is  as  follows: 

Via  an  ARPANET  RJE  proeess,  a  user  connects  his 
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Queue  of 
Submitted  Jobs 


Figure  9 — Submission  of  RJE  input 


terminal  to  an  RJE  server  process  at  the  HOST  to 
which  he  intends  to  submit  his  job  “deck.  Through 
a  short  dialogue,  he  establishes  the  source  of  his  input 


and  initiates  its  transfer  using  the  File  Transfer  Proto¬ 
col.  At  some  later  time,  the  user  reconnects  to  the  ap¬ 
propriate  RJE  server  and  makes  an  inquiry  on  the 


Figure  10—  Retrieval  of  RJE  output 
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status  of  Iris  job.  When  notified  that  his  iuput  lias  been 
processed,  he  then  issues  commands  to  the  serving 
HOST  to  transfer  his  output  back. 

We  can  of  course  imagine  more  automatic  ways  of 
achieving  these  same  functions.  A  user  might  need  only 
type  a  job  submission  command  to  his  local  system. 
Automatically  and  invisibly,  then,  the  local  system 
would  couucct  and  converse  with  the  specified  RJF, 
server  causing  the  desired  output  to  later  appear  in  the 
users  file  area  or  perhaps  on  a  local  line  printer.  The 
intention  is  to  design  the  RJE  protocol  so  that  the  ex¬ 
plicit  use  can  start  immediately  and  the  more  automatic 
RJE  systems  can  be  built  as  desired. 

OTHER  PROTOCOLS  AND  CONCLUSIONS 

One  of  the  more  difficult  problems  in  utilizing  a  net¬ 
work  of  diverse  computers  and  operating  systems  is  that 
of  dealing  with  incompatible  data  streams.  Computers 
and  their  language  processors  have  many  way’s  of 
representing  data.  To  make  use  of  different  computers 
it  is  necessary  to  (1)  produce  a  mediation  scheme  for 
each  incompatibility  or  (2)  produce  a  standard  repre¬ 
sentation.  There  arc  many  strong  arguments  for  a 
standard  representation,  but  it  has  been  hypothesized 
that  if  there  were  a  simple  way  of  expressing  a  limited 
set  of  transformations  on  data  streams,  that  a  large 
number  of  incompatibilities  could  be  resolved  and  a 
great  deal  of  computer-computer  cooperation  expedited. 

The  bulk  of  protocol  work  is  being  done  with  the 
invention  of  standard  representations.  The  TELNET 
protocol,  as  discussed,  is  founded  on  the  notion  of  a 
standard  terminal  called  the  Network  Virtual  Terminal 
(NVT).  The  File  Transfer  Protocol  is  working  toward 
a  standard  sequential  file  (a  Network  Virtual  File?). 
So  it  is  also  with  less  advanced  protocol  work  in  graphics 
and  data  management. 

There  Is  one  experiment  which  is  taking  the  trans¬ 
formational  approach  to  dealing  with  incompatibilities. 
The  Data  Reconfiguration  Service  (DRS)  is  to  be 
generally  available  for  mediating  between  incompatible 
stream  configurations  as  directed  by  user-supplied 
transformations.11 
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